Bug 2231633

Summary: [openvswitch3.2 pre-test] [ovs-dpdk] E810 ice card reports many dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Rick Alongi <ralongi>
Component: openvswitch3.1Assignee: OVS Triage <ovs-triage>
Status: NEW --- QA Contact: Rick Alongi <ralongi>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: FDP 23.FCC: ctrautma, jhsiao, ralongi, zfang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ovs log files from client system none

Description Rick Alongi 2023-08-12 19:05:36 UTC 
Created attachment 1983150 [details]
ovs log files from client system

Description of problem:

Multiple different VM based tests fail as VMs cannot ping each other.

Hundreds of these messages in ovs-vswitchd.log:
2023-08-11T23:02:07.634Z|00017|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event

Version-Release number of selected component (if applicable):

RHEL-9.2.0-updates-20230809.16
kernel version: 5.14.0-284.27.1.el9_2.x86_64

[root@netqe44 openvswitch]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-34.el9fdp.noarch
openvswitch3.2-3.2.0-0.2.el9fdp.x86_64
python3-openvswitch3.2-3.2.0-0.2.el9fdp.x86_64
openvswitch3.2-test-3.2.0-0.2.el9fdp.noarch

How reproducible:


Steps to Reproduce:
1. Observed in automated topology tests
2.
3.

Actual results:

VMs cannot ping each other nor send traffic.

Expected results:

VMs can successfully ping each other and send traffic.

Additional info:

Problem only happens with the tests referenced below and only in ovs-dpdk environment:

ovs_test_vm_nic ovs_test_vm_multiple_vlans ovs_test_vm_vlan_id0 ovs_test_vm_gre1_ipv6 ovs_test_vm_gre_flow_ipv6 ovs_test_vm_vlan_gre1_ipv6 ovs_test_vlan_mode_access_vm ovs_test_vlan_mode_trunk_vm ovs_test_vlan_mode_native_untagged_vm ovs_test_vlan_mode_native_tagged_vm

- Same tests run with same versions against Mellanox ConnectX6 DX card (mlx5_core) show no problem.
- Same tests run against E810 ice card pass 100% using openvswitch3.1 from FDP 23.F with RHEL-9.2.
- This issue is also seen on these same tests using openvswitch3.2 with RHEL-9.3.

Note: VMs are running RHEL-8.6 instead of RHEL-9.2 due to BZ https://bugzilla.redhat.com/show_bug.cgi?id=2184976 but problem is also seen using RHEL-9.2 VM with RHEL-9.3 which contains the fix for https://bugzilla.redhat.com/show_bug.cgi?id=2184976.

sos reports can be found here:

http://netqe-infra01.knqe.lab.eng.bos.redhat.com/sosreports/sosreport-netqe31-2023-08-12-ejsinsu.tar.xz
http://netqe-infra01.knqe.lab.eng.bos.redhat.com/sosreports/sosreport-netqe44-2023-08-12-pxttxtk.tar.xz

Beaker job links:

openvswitch3.2, RHEL-9.2: https://beaker.engineering.redhat.com/jobs/8180530
openvswitch3.2, RHEL-9.3: https://beaker.engineering.redhat.com/jobs/8182138
openvswitch3.1, RHEL-9.2: https://beaker.engineering.redhat.com/jobs/8089673