Bug 1700906 - PVP Results for Open vSwitch on Broadcom Hardware on RHEL8.0
Summary: PVP Results for Open vSwitch on Broadcom Hardware on RHEL8.0
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch
Version: FDP 19.E
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: qding
QA Contact: qding
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-17 15:55 UTC by raghavendrachari.kamsali
Modified: 2023-07-13 07:25 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
PVP Results for Open vSwitch with DPDK Datapath L2 10hrs (6.93 MB, application/gzip)
2019-04-17 15:55 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for Open vSwitch with DPDK Datapath L3 10hrs (6.89 MB, application/gzip)
2019-04-17 15:58 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for Open vSwitch Kernel Datapath L3 10hrs (9.27 MB, application/gzip)
2019-04-17 16:00 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for Open vSwitch with DPDK Datapath L2 1hrs (1.59 MB, application/gzip)
2019-04-17 16:04 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for Open vSwitch with DPDK Datapath L3 1hrs (1.45 MB, application/gzip)
2019-04-19 07:03 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for Open vSwitch with Kernel Datapath L2 1hrs (1.38 MB, application/gzip)
2019-04-19 07:04 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for Open vSwitch Kernel Datapath L3 1hrs (1.42 MB, application/gzip)
2019-04-24 13:49 UTC, raghavendrachari.kamsali
no flags Details
OVS FUnctional verification RHEL8.0 Client result (9.21 MB, application/x-tar)
2019-05-01 14:42 UTC, raghavendrachari.kamsali
no flags Details
OVS Functional verification RHEL8.0 Server Result log (7.29 MB, application/x-tar)
2019-05-01 14:45 UTC, raghavendrachari.kamsali
no flags Details
OVS Functional verification RHEL8.0 Client Rerun Result log (8.61 MB, application/x-tar)
2019-05-13 05:59 UTC, raghavendrachari.kamsali
no flags Details
PVP Results for OpenvSwitch 2.11 with DPDK datapath (6.33 MB, application/gzip)
2019-06-04 07:09 UTC, raghavendrachari.kamsali
no flags Details
server results (7.24 MB, application/x-tar)
2019-06-14 08:05 UTC, Davide Caratti
no flags Details
client results (10.56 MB, application/x-tar)
2019-06-14 08:06 UTC, Davide Caratti
no flags Details
PVP Results for Open vSwitch with DPDK for ovs2.11 (6.31 MB, application/gzip)
2019-08-02 11:23 UTC, raghavendrachari.kamsali
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-22 0 None None None 2021-09-21 10:03:46 UTC

Description raghavendrachari.kamsali 2019-04-17 15:55:31 UTC
Created attachment 1555946 [details]
PVP Results for Open vSwitch with DPDK Datapath L2 10hrs

This bug will serve as a landing spot for the latest Broadcom PVP test results on RHEL8.0

Comment 1 raghavendrachari.kamsali 2019-04-17 15:58:59 UTC
Created attachment 1555947 [details]
PVP Results for Open vSwitch with DPDK Datapath L3 10hrs

Comment 2 raghavendrachari.kamsali 2019-04-17 16:00:23 UTC
Created attachment 1555948 [details]
PVP Results for Open vSwitch Kernel Datapath L3 10hrs

Comment 3 raghavendrachari.kamsali 2019-04-17 16:04:35 UTC
Created attachment 1555951 [details]
PVP Results for Open vSwitch with DPDK Datapath L2 1hrs

Comment 4 raghavendrachari.kamsali 2019-04-19 07:03:20 UTC
Created attachment 1556350 [details]
PVP Results for Open vSwitch with DPDK Datapath L3 1hrs

Comment 5 raghavendrachari.kamsali 2019-04-19 07:04:37 UTC
Created attachment 1556351 [details]
PVP Results for Open vSwitch with Kernel Datapath L2 1hrs

Comment 6 raghavendrachari.kamsali 2019-04-24 13:49:40 UTC
Created attachment 1558209 [details]
PVP Results for Open vSwitch Kernel Datapath L3 1hrs

Comment 7 raghavendrachari.kamsali 2019-05-01 14:42:41 UTC
Created attachment 1560830 [details]
OVS FUnctional verification RHEL8.0 Client result

Comment 8 raghavendrachari.kamsali 2019-05-01 14:45:04 UTC
Created attachment 1560833 [details]
OVS Functional verification RHEL8.0 Server Result log

Comment 9 Christian Trautman 2019-05-01 22:02:59 UTC
Hello the client side of functional tests all report failures.

Can you please provide the rh_nic_cert.sh from both the client and server side so I can see how it was configured?

Thanks,

Comment 10 Christian Trautman 2019-05-03 13:22:22 UTC
We have seen in the past where if all tests fail then the switch could be a problem.  You can also try by doing the test with a back to back connection.  The bonding tests will fail, but we can deal with that after.

Comment 11 qding 2019-05-05 02:45:43 UTC
From log below in attachment 1560830 [details], the port on host wsfd-netdev85.ntdv.lab.eng.bos.redhat.com is in state NO-CARRIER. Please check the connection between enp5s0f0np0 and the switch port.

100: enp5s0f0np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN mode DEFAULT group default qlen 1000
link/ether 00:0a:f7:b6:e0:c0 brd ff:ff:ff:ff:ff:ff promiscuity 1 
openvswitch_slave addrgenmode none numtxqueues 74 numrxqueues 74 gso_max_size 65536 gso_max_segs 65535 portname p0

Comment 12 raghavendrachari.kamsali 2019-05-13 05:59:37 UTC
Created attachment 1567727 [details]
OVS Functional verification RHEL8.0 Client Rerun Result log

OVS Functional verification RHEL8.0 Client Rerun Result log

Comment 13 raghavendrachari.kamsali 2019-05-13 06:06:44 UTC
Can you please let me know how can i upload a zip file more than 30MB???

Comment 14 qding 2019-05-13 13:24:03 UTC
from log in attachment 1567727 [details] in comment#12, they're all test for multicast, but only vlan related test failed. Because I don't see switch configuration, please check if switch has the correct setting. And please note that usually several tagged frames will be used and keep 3~100 vlan IDs are permitted.

Comment 15 raghavendrachari.kamsali 2019-05-13 13:58:05 UTC
Hi, 

The test suite is hanging at 'ovs-vlan test 9000 ' from the past 2days,
So i did CTRL+C after that it executed some test , I collected the logs and shared to this BZ.
Could you please tell me is there any way to run the failed test cases individually.


Regards,
Raghav

Comment 16 qding 2019-05-14 02:05:41 UTC
(In reply to raghavendrachari.kamsali from comment #15)
> Hi, 
> 
> The test suite is hanging at 'ovs-vlan test 9000 ' from the past 2days,
> So i did CTRL+C after that it executed some test , I collected the logs and
> shared to this BZ.
> Could you please tell me is there any way to run the failed test cases
> individually.
> 

With the example for running only ovs_test_vlan which is in code topo.sh. export QE_TEST=ovs_test_vlan in rh_nic_cert.sh or the running shell on both machines

And please find the script file rh_nic_cert.sh and go to the bottom, the code below can be found, then comment out lines of 130 and 132

129 pushd openvswitch
130 grep 'bz1468631' <<<"${QE_SKIP_TEST}" || . bz1468631.sh |& tee -a ${test_log}
131 . topo.sh |& tee -a ${test_log}
132 . mcast_snoop.sh |& tee -a ${test_log}
133 popd &>/dev/null
134

Comment 17 raghavendrachari.kamsali 2019-05-15 13:40:40 UTC
Hi,

I followed the above comment & rerun the test , but same isse happened , test hangs at ovs_vlan_test 9000 .
Regards,
Raghav

Comment 18 qding 2019-05-16 01:44:22 UTC
(In reply to raghavendrachari.kamsali from comment #17)
> Hi,
> 
> I followed the above comment & rerun the test , but same isse happened ,
> test hangs at ovs_vlan_test 9000 .

if ovs_vlan_test passed and only "ovs_vlan_test 9000" failed, please check if you have set mtu to 9216 in switch.

Comment 22 Davide Caratti 2019-05-23 08:28:57 UTC
hi Raghav,

looking at the logs, we noticed:

07:52:36[DEBUG   ][dutSh.10.13.241.98]: STDOUT: >>ovs-vswitchd (Open vSwitch) 2.10.0
DPDK 17.11.4

probably the test was done using a brew build of openvswitch that included the fix for IOVA VA mode (bz1645523).
Since RHEL8 will be using openvswitch2.11 (and dpdk 18.11), can you please provide results for the PVP using an official RPM (e.g. the one released with RHEA-2019:40489-02 https://errata.devel.redhat.com/advisory/40366)?

thanks you in advance!
-- 
davide

Comment 23 raghavendrachari.kamsali 2019-06-04 07:09:34 UTC
Created attachment 1576921 [details]
PVP Results for OpenvSwitch 2.11 with DPDK datapath

PVP Results for OpenvSwitch 2.11 with DPDK datapath

Comment 24 Davide Caratti 2019-06-10 13:52:59 UTC
(In reply to qding from comment #18)
> (In reply to raghavendrachari.kamsali from comment #17)
> > Hi,
> > 
> > I followed the above comment & rerun the test , but same isse happened ,
> > test hangs at ovs_vlan_test 9000 .
> 
> if ovs_vlan_test passed and only "ovs_vlan_test 9000" failed, please check
> if you have set mtu to 9216 in switch.

hello Qi Jun Ding / Christian,

I tried this on netdev84 / 85,  the MTU to 9216 on the Virtual Chassis, but the following command

 # ping 192.0.2.255 -b -c1 -Mdo -s$SZ -I enp5s0f0np0

it's possible to see forwarded packets only when $SZ is less or equal to 1490.
Do we have a command reference for srtting up VSPERF on Virtual Chassis?
thanks a lot!
-- 
davide

Comment 25 Davide Caratti 2019-06-10 13:55:57 UTC
(In reply to Davide Caratti from comment #24)
> (In reply to qding from comment #18)
> > (In reply to raghavendrachari.kamsali from comment #17)
> Do we have a command reference for srtting up VSPERF on Virtual Chassis?

(eraly shot) the question is: do we have a command reference for preconfiguring the Virtual Chassis with the Functional Testsuite?

thanks!
-- 
davide

Comment 26 qding 2019-06-11 01:31:08 UTC
(In reply to Davide Caratti from comment #25)
> (In reply to Davide Caratti from comment #24)
> > (In reply to qding from comment #18)
> > > (In reply to raghavendrachari.kamsali from comment #17)
> > Do we have a command reference for srtting up VSPERF on Virtual Chassis?
> 
> (eraly shot) the question is: do we have a command reference for
> preconfiguring the Virtual Chassis with the Functional Testsuite?
> 

You're talking Virtual Chassis for JunOS, right? No preconfiguration for it.

Comment 27 Davide Caratti 2019-06-14 08:05:57 UTC
Created attachment 1580589 [details]
server results

Comment 28 Davide Caratti 2019-06-14 08:06:50 UTC
Created attachment 1580590 [details]
client results

Comment 29 raghavendrachari.kamsali 2019-08-02 11:23:27 UTC
Created attachment 1599991 [details]
PVP Results for Open vSwitch with DPDK for ovs2.11

Comment 30 Davide Caratti 2019-08-20 14:53:37 UTC
(In reply to raghavendrachari.kamsali from comment #29)
> Created attachment 1599991 [details]
> PVP Results for Open vSwitch with DPDK for ovs2.11

apparently the test was interrupted before running the 10-hours batch, both for L2 and L3 cases. Looking at the logs, I see this:

13:29:53[DEBUG   ][dutSh.10.13.241.98]: STDOUT: >>EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:02.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1000 net_virtio
EAL:   using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(0)
PANIC in main():
Empty set of forwarding logical cores - check the core mask supplied in the command parameters
5: [testpmd(_start+0x2e) [0x42f48e]]
4: [/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f690b250813]]
3: [testpmd() [0x42e6c8]]
2: [/lib64/librte_eal.so.9(__rte_panic+0xc1) [0x7f690c456f37]]
1: [/lib64/librte_eal.so.9(rte_dump_stack+0x32) [0x7f690c463752]]
<<END
13:29:53[DEBUG   ][dutSh.10.13.241.98]: STDERR: >>Warning: Permanently added '192.168.122.210' (ECDSA) to the list of known hosts.
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: RTE Version: 'DPDK 18.11.0'
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
<<END
13:29:53[INFO    ][script]: - Get OpenFlow and DataPath port numbers..

so it seems that testpmd is doing segmentation fault in the guest (probably because of wrong CPU pinning?)

Comment 31 raghavendrachari.kamsali 2019-09-05 14:50:31 UTC
Now Provided cpu cores on same numa node on which interface installed.


Note You need to log in before you can comment on or make changes to this bug.