Bug 2131339

Summary:	CVE-2022-4269 kernel: net: CPU soft lockup in TC mirred egress-to-ingress action [rhel-8.8.0]
Product:	Red Hat Enterprise Linux 8	Reporter:	William Zhao <wizhao>
Component:	kernel	Assignee:	Davide Caratti <dcaratti>
kernel sub component:	Networking	QA Contact:	Li Shuang <shuali>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	bnemeth, dcaratti, jiji, kzhang, lariel, mcascell, mleitner, shuali
Version:	8.6	Keywords:	Security, SecurityTracking, Triaged, ZStream
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	kernel-4.18.0-456.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2164648 2164649 2164650 (view as bug list)		Environment:
Last Closed:	2023-05-16 08:52:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2099862, 2099873, 2110018, 2130221, 2131355, 2150272, 2150276, 2164648, 2164649, 2164650

Description William Zhao 2022-09-30 17:58:49 UTC

Description of problem:

In an Openshift 4.11 environment with Nvidia Mlx BF2 in NIC mode. Using the hardware offload capabilities of the NIC. During certain flows namely those where 2 host backed pods are used to communicate with each other, the TCP connection would soft lock-up. This was seem in these 2 flows:
10-a: Host Pod -> Cluster IP Service traffic (Host Backend - Same Node).
12-a: Host Pod -> NodePort Service traffic (Host Backend - Same Node).

Version-Release number of selected component (if applicable):
4.18.0-372.30.1.el8_6

How reproducible:
Run Iperf with these two flows:
10-a: Host Pod -> Cluster IP Service traffic (Host Backend - Same Node).
12-a: Host Pod -> NodePort Service traffic (Host Backend - Same Node).

Or run the reproducer from Davide:
#!/bin/bash

do_setup() {
        local i client0_l2addr server0_l2addr client1_l2addr server1_l2addr

        ip netns add ns0
        ip netns add ns1
        ip -n ns0 link add name client0 type veth peer name client1 netns ns1
        ip -n ns0 link add name server0 type veth peer name server1 netns ns1

        for i in 0 1 ; do
                ip -n ns$i link set dev lo up
                ip -n ns$i link set dev client$i up
                ip -n ns$i link set dev server$i up

                ip -n ns$i a a dev client$i 192.0.2.$((i+1))/24
                ip -n ns$i a a dev server$i 198.51.100.$((i+1))/24
		if [ $i -eq 0 ]; then
	                tc -n ns$i qdisc add dev client$i clsact
        	        tc -n ns$i qdisc add dev server$i clsact
		fi
		# useless use of eval+cat
		eval "client${i}_l2addr=`ip netns exec ns${i} cat /sys/class/net/client${i}/address`"
		eval "server${i}_l2addr=`ip netns exec ns${i} cat /sys/class/net/server${i}/address`"
        done

	# stateful SNAT + DNAT on client0 egress
	tc -n ns0 filter add dev client0 egress prio 1 chain 0 proto ip flower ip_proto tcp \
		action ct commit nat src addr 198.51.100.2 pipe \
		action ct clear pipe \
		action ct commit nat dst addr 198.51.100.1 pipe \
		action goto chain 3

	# forwarding
	tc -n ns0 filter add dev client0 egress prio 1 chain 3 proto ip flower ip_proto tcp \
		action ct clear pipe \
		action pedit ex munge eth dst set $server0_l2addr pipe \
		action pedit ex munge eth src set $server1_l2addr pipe \
		action csum ip and tcp pipe \
		action skbedit ptype host pipe \
		action mirred ingress redirect dev server0


	# stateful SNAT + DNAT on server0 egress
	tc -n ns0 filter add dev server0 egress prio 1 chain 0 proto ip flower ip_proto tcp \
		action ct commit nat src addr 192.0.2.2 pipe \
		action ct clear pipe \
		action ct commit nat dst addr 192.0.2.1 pipe \
		action goto chain 3
	# forwarding
	tc -n ns0 filter add dev server0 egress prio 1 chain 3 proto ip flower ip_proto tcp \
		action pedit ex munge eth dst set $client0_l2addr pipe \
		action pedit ex munge eth src set $client1_l2addr pipe \
		action csum ip and tcp pipe \
		action skbedit ptype host pipe \
		action mirred ingress redirect dev client0

}


do_cleanup() {
        local i

        for i in ns0 ns1 ; do
                ip netns del $i
        done >/dev/null 2>&1
}

do_test1() {
	 ip netns exec ns0 iperf3 -s -D
	 ip netns exec ns0 iperf3 -c 192.0.2.2 -i1 -t5
	 ip netns exec ns0 pkill iperf3
	 :
}

do_test2() {
	local i
	ip netns exec ns0 ncat -4 -i 15 -c "sleep 5" -C  -o /tmp/repro  -l 198.51.100.1 4321  &
	for i in `seq 1 2`; do
		printf "hello world ($i)\n" 
		sleep 1
	done | ip netns exec ns0 ncat 192.0.2.2 4321
	:
}

do_test3() {
	ip netns exec ns0 nc -l -4 -k &
	sleep 1
	echo "hello world" | ip netns exec ns0 nc -4 192.0.2.2
	sleep 1
	pkill nc
}

case ${1:-nocmd} in
cleanup|setup|test*) cmd=$1; shift ; eval "do_${cmd} $@" ;;
*) printf "usage: $0 <cleanup|setup|test1|test2|test3>\n" ;;
esac

Actual results:
Soft lockup occurs with the following trace visible in:
https://bugzilla.redhat.com/show_bug.cgi?id=2110018#c18

Expected results:
No soft-lock up occurs.

Comment 1 William Zhao 2022-09-30 18:01:18 UTC

Davide in fact has a reproducer and has found a possible fix

Proposed patch and discussions: https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/

Comment 2 Davide Caratti 2022-10-06 09:10:12 UTC

(In reply to William Zhao from comment #1)
> Davide in fact has a reproducer and has found a possible fix
> 
> Proposed patch and discussions:
> https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.
> 1663945716.git.dcaratti/

the fix in comment #1 has been substantially nacked because it changes the behavior of TC ingress. A second attempt has been proposed upstream ( https://lore.kernel.org/netdev/YzxwCy7R0MdWZuO4@dcaratti.users.ipa.redhat.com/ )

Comment 5 William Zhao 2022-11-14 16:37:47 UTC

https://bugzilla.redhat.com/show_bug.cgi?id=2110018#c18 may not be accessible by the public. Copying the kernel panic here:

Kernel Panic for flow 10-a: Host Pod -> Cluster IP Service traffic (Host Backend - Same Node).
[ 1032.467724] watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [swapper/26:0]
[ 1032.476191] Modules linked in: act_tunnel_key nf_conntrack_netlink veth act_csum act_pedit act_ct nf_flow_table xt_addrtype ipt_REJECT nf_reject_ipv4 xt_nat vhost_net vhost tap tun xt_CT xt_REDIRECT ip6t_MASQUERADE ipt_MASQUERADE xt_conntrack xt_comment nft_counter xt_mark nft_compat nft_chain_nat nf_tables act_skbedit act_mirred geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink act_gact cls_flower sch_ingress openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mlx5_vdpa vringh vhost_iotlb vdpa overlay ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif kvm_intel mgag200 rpcrdma drm_kms_helper iTCO_wdt kvm dell_smbios syscopyarea irqbypass iTCO_vendor_support wmi_bmof dell_wmi_descriptor dcdbas sysfillrect rapl sunrpc sysimgblt fb_sys_fops intel_cstate mei_me acpi_ipmi drm pcspkr intel_uncore mei i2c_i801 lpc_ich wmi ipmi_si acpi_power_meter ip_tables xfs
[ 1032.476281]  libcrc32c
[ 1032.522725] watchdog: BUG: soft lockup - CPU#38 stuck for 22s! [swapper/38:0]
[ 1032.574890]  rdma_ucm
[ 1032.578141] Modules linked in:
[ 1032.586858]  ib_srpt
[ 1032.590028]  act_tunnel_key
[ 1032.594632]  ib_isert
[ 1032.597680]  nf_conntrack_netlink
[ 1032.601972]  iscsi_target_mod
[ 1032.605087]  veth
[ 1032.609877]  target_core_mod
[ 1032.613675]  act_csum act_pedit
[ 1032.617061]  ib_iser
[ 1032.620766]  act_ct
[ 1032.625349]  rdma_cm
[ 1032.628353]  nf_flow_table
[ 1032.631879]  iw_cm
[ 1032.634877]  xt_addrtype
[ 1032.639013]  ib_umad
[ 1032.641830]  ipt_REJECT
[ 1032.645762]  ib_ipoib
[ 1032.648727]  nf_reject_ipv4
[ 1032.652510]  ib_cm
[ 1032.655526]  xt_nat
[ 1032.659591]  mlx5_ib
[ 1032.662310]  vhost_net
[ 1032.665597]  ib_uverbs
[ 1032.668445]  vhost
[ 1032.671924]  ib_core
[ 1032.674909]  tap
[ 1032.677971]  sd_mod
[ 1032.680731]  tun
[ 1032.683544]  t10_pi
[ 1032.686181]  xt_CT
[ 1032.688920]  sg
[ 1032.691517]  xt_REDIRECT
[ 1032.694353]  crct10dif_pclmul
[ 1032.696563]  ip6t_MASQUERADE
[ 1032.699854]  crc32_pclmul
[ 1032.703234]  ipt_MASQUERADE
[ 1032.706798]  crc32c_intel
[ 1032.709809]  xt_conntrack
[ 1032.713237]  mlx5_core
[ 1032.716198]  xt_comment
[ 1032.719384]  ahci
[ 1032.722050]  nft_counter xt_mark
[ 1032.724990]  libahci
[ 1032.727176]  nft_compat
[ 1032.730836]  ixgbe
[ 1032.733250]  nft_chain_nat
[ 1032.736065]  igb
[ 1032.738276]  nf_tables
[ 1032.741320]  psample
[ 1032.743354]  act_skbedit act_mirred
[ 1032.746035]  ghash_clmulni_intel
[ 1032.748414]  geneve ip6_udp_tunnel
[ 1032.752230]  i2c_algo_bit
[ 1032.755652]  udp_tunnel
[ 1032.759380]  mlxfw
[ 1032.762196]  nfnetlink_cttimeout
[ 1032.764975]  libata
[ 1032.767178]  nfnetlink
[ 1032.770733]  megaraid_sas
[ 1032.773020]  act_gact cls_flower
[ 1032.775709]  pci_hyperv_intf
[ 1032.778524]  sch_ingress
[ 1032.782091]  mdio
[ 1032.785169]  openvswitch
[ 1032.788041]  dca
[ 1032.790159]  nf_conncount
[ 1032.793029]  dm_multipath
[ 1032.795066]  nf_nat nf_conntrack
[ 1032.798021]  dm_mirror
[ 1032.800839]  nf_defrag_ipv6
[ 1032.804411]  dm_region_hash
[ 1032.806969]  nf_defrag_ipv4
[ 1032.810115]  dm_log
[ 1032.813117]  mlx5_vdpa vringh
[ 1032.816264]  dm_mod
[ 1032.818570]  vhost_iotlb
[ 1032.821892]  be2iscsi
[ 1032.824197]  vdpa overlay
[ 1032.827091]  bnx2i
[ 1032.829573]  ext4
[ 1032.832562]  cnic
[ 1032.834785]  mbcache
[ 1032.837069]  uio
[ 1032.839209]  jbd2 intel_rapl_msr
[ 1032.841753]  cxgb4i
[ 1032.843800]  intel_rapl_common
[ 1032.847381]  cxgb4
[ 1032.849686]  isst_if_common
[ 1032.853097]  tls
[ 1032.855319]  skx_edac
[ 1032.858472]  libcxgbi
[ 1032.860517]  nfit
[ 1032.863147]  libcxgb
[ 1032.865630]  libnvdimm
[ 1032.867908]  qla4xxx
[ 1032.870301]  x86_pkg_temp_thermal
[ 1032.873011]  iscsi_boot_sysfs
[ 1032.875403]  intel_powerclamp
[ 1032.879075]  iscsi_tcp
[ 1032.882243]  coretemp
[ 1032.885564]  libiscsi_tcp
[ 1032.888124]  ipmi_ssif kvm_intel
[ 1032.890748]  libiscsi
[ 1032.893567]  mgag200
[ 1032.897149]  scsi_transport_iscsi
[ 1032.899624]  rpcrdma
[ 1032.902150]  ipmi_devintf
[ 1032.905667]  drm_kms_helper
[ 1032.908206]  ipmi_msghandler
[ 1032.911031]  iTCO_wdt
[ 1032.914174]  fuse
[ 1032.917254]  kvm
[ 1032.919877] 
[ 1032.922003]  dell_smbios
[ 1032.924188] Red Hat flags:
[ 1032.925882]  syscopyarea irqbypass
[ 1032.928753]  eBPF/cgroup
[ 1032.931652]  iTCO_vendor_support
[ 1032.935398] 
[ 1032.938128]  wmi_bmof
[ 1032.941700] CPU: 26 PID: 0 Comm: swapper/26 Tainted: G          I  L   --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[ 1032.943390]  dell_wmi_descriptor
[ 1032.946001] Hardware name: Dell Inc. PowerEdge R640/0X45NX, BIOS 2.8.2 08/27/2020
[ 1032.956821]  dcdbas
[ 1032.960443] RIP: 0010:native_queued_spin_lock_slowpath+0x5b/0x1b0
[ 1032.968165]  sysfillrect rapl
[ 1032.970693] Code: 6d f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 47 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00
[ 1032.977047]  sunrpc
[ 1032.980472] RSP: 0018:ffffb4818d063e18 EFLAGS: 00000202
[ 1032.999834]  sysimgblt
[ 1033.002474]  ORIG_RAX: ffffffffffffff13
[ 1033.008023]  fb_sys_fops
[ 1033.010943] RAX: 00000000009c0101 RBX: ffff981bd1376900 RCX: 0000000000000020
[ 1033.015105]  intel_cstate
[ 1033.018202] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff981cfba15288
[ 1033.025668]  mei_me
[ 1033.028885] RBP: ffff981cfba15200 R08: 0000000000000001 R09: 0000000000000000
[ 1033.036372]  acpi_ipmi drm
[ 1033.039101] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1033.046614]  pcspkr
[ 1033.049990] R13: ffff981cfba15288 R14: ffff9833018728f0 R15: ffff9833018728dc
[ 1033.057520]  intel_uncore
[ 1033.060322] FS:  0000000000000000(0000) GS:ffff983260d40000(0000) knlGS:0000000000000000
[ 1033.067871]  mei
[ 1033.071227] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1033.079757]  i2c_i801
[ 1033.082372] CR2: 00007f73c2d34948 CR3: 00000024f9410004 CR4: 00000000007706e0
[ 1033.088573]  lpc_ich
[ 1033.091642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1033.099247]  wmi
[ 1033.102267] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1033.109890]  ipmi_si
[ 1033.112595] PKRU: 55555554
[ 1033.120497]  acpi_power_meter
[ 1033.122859] mlx5_cmd_check: 1160 callbacks suppressed
[ 1033.122862] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.122962] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.123059] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.123155] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.123255] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.123580] Call Trace:
[ 1033.123706] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.123998] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.124283] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.124378] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.124475] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14733): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1033.126803]  ip_tables
[ 1033.130663]  <IRQ>
[ 1033.136234]  xfs
[ 1033.151779]  _raw_spin_lock+0x1a/0x20
[ 1033.166595]  libcrc32c
[ 1033.182429]  tcp_v4_rcv+0xad5/0xc50
[ 1033.197396]  rdma_ucm ib_srpt
[ 1033.213493]  ip_protocol_deliver_rcu+0x2c/0x1d0
[ 1033.216645]  ib_isert
[ 1033.232800]  ip_local_deliver_finish+0x4d/0x60
[ 1033.247913]  iscsi_target_mod
[ 1033.264216]  ip_local_deliver+0xe0/0xf0
[ 1033.279398]  target_core_mod
[ 1033.295799]  ? ip_protocol_deliver_rcu+0x1d0/0x1d0
[ 1033.298944]  ib_iser
[ 1033.302323]  ip_rcv+0x27b/0x36f
[ 1033.304941]  rdma_cm
[ 1033.309955]  ? inet_add_protocol.cold.1+0x1e/0x1e
[ 1033.313085]  iw_cm
[ 1033.317904]  __netif_receive_skb_core+0x5cd/0xcb0
[ 1033.321623]  ib_umad ib_ipoib
[ 1033.327465]  ? skb_clone+0x4d/0xb0
[ 1033.330476]  ib_cm
[ 1033.336211]  ? kmem_cache_alloc+0x13f/0x280
[ 1033.339909]  mlx5_ib
[ 1033.345010]  netif_receive_skb_internal+0x3d/0xb0
[ 1033.348604]  ib_uverbs
[ 1033.354634]  ? __skb_clone+0x29/0x130
[ 1033.357520]  ib_core
[ 1033.361878]  tcf_mirred_act+0x241/0x510 [act_mirred]
[ 1033.364758]  sd_mod
[ 1033.370664]  tcf_action_exec+0x77/0x120
[ 1033.373355]  t10_pi
[ 1033.379248]  fl_classify+0x1e6/0x1f0 [cls_flower]
[ 1033.382887]  sg
[ 1033.387455]  ? nf_conntrack_tcp_packet+0x33e/0xb90 [nf_conntrack]
[ 1033.390114]  crct10dif_pclmul
[ 1033.395447]  tcf_classify+0x79/0x270
[ 1033.398293]  crc32_pclmul
[ 1033.404156]  __dev_queue_xmit+0x30e/0xa40
[ 1033.407165]  crc32c_intel
[ 1033.411928]  ? nf_route_table_hook4+0x8e/0x120 [nf_tables]
[ 1033.414724]  mlx5_core
[ 1033.420731]  ip_finish_output2+0x269/0x430
[ 1033.423405]  ahci libahci
[ 1033.428238]  ip_output+0x70/0xe0
[ 1033.430896]  ixgbe
[ 1033.436525]  ? __ip_finish_output+0x1c0/0x1c0
[ 1033.438795]  igb
[ 1033.445773]  __ip_queue_xmit+0x15d/0x430
[ 1033.449267]  psample
[ 1033.453703]  __tcp_transmit_skb+0x9fd/0xb50
[ 1033.456800]  ghash_clmulni_intel
[ 1033.461611]  tcp_write_xmit+0x43d/0x12c0
[ 1033.464668]  i2c_algo_bit
[ 1033.470901]  ? tcp_rcv_established+0x2c5/0x5c0
[ 1033.473669]  mlxfw
[ 1033.478461]  __tcp_push_pending_frames+0x32/0xf0
[ 1033.481461]  libata
[ 1033.485317]  tcp_rcv_established+0x2fb/0x5c0
[ 1033.487671]  megaraid_sas
[ 1033.492626]  ? tcp_v4_inbound_md5_hash+0x67/0x140
[ 1033.494812]  pci_hyperv_intf
[ 1033.499344]  tcp_v4_do_rcv+0x12a/0x1e0
[ 1033.501881]  mdio
[ 1033.506680]  tcp_v4_rcv+0xb43/0xc50
[ 1033.510256]  dca
[ 1033.514795]  ip_protocol_deliver_rcu+0x2c/0x1d0
[ 1033.517771]  dm_multipath dm_mirror
[ 1033.522829]  ip_local_deliver_finish+0x4d/0x60
[ 1033.525192]  dm_region_hash
[ 1033.530431]  ip_local_deliver+0xe0/0xf0
[ 1033.532887]  dm_log
[ 1033.537774]  ? ip_protocol_deliver_rcu+0x1d0/0x1d0
[ 1033.540744]  dm_mod
[ 1033.546067]  ip_rcv+0x27b/0x36f
[ 1033.549310]  be2iscsi
[ 1033.553681]  ? inet_add_protocol.cold.1+0x1e/0x1e
[ 1033.555966]  bnx2i
[ 1033.560069]  __netif_receive_skb_core+0x5cd/0xcb0
[ 1033.562264]  cnic
[ 1033.567411]  ? skb_clone+0x4d/0xb0
[ 1033.571253]  uio
[ 1033.576315]  ? kmem_cache_alloc+0x13f/0x280
[ 1033.579460]  cxgb4i
[ 1033.583913]  netif_receive_skb_internal+0x3d/0xb0
[ 1033.586366]  cxgb4
[ 1033.591774]  ? __skb_clone+0x29/0x130
[ 1033.594247]  tls
[ 1033.598003]  tcf_mirred_act+0x241/0x510 [act_mirred]
[ 1033.600629]  libcxgbi
[ 1033.605947]  tcf_action_exec+0x77/0x120
[ 1033.608308]  libcxgb
[ 1033.613635]  fl_classify+0x1e6/0x1f0 [cls_flower]
[ 1033.615914]  qla4xxx
[ 1033.619931]  ? nf_conntrack_tcp_packet+0x33e/0xb90 [nf_conntrack]
[ 1033.622107]  iscsi_boot_sysfs
[ 1033.626891]  tcf_classify+0x79/0x270
[ 1033.629335]  iscsi_tcp
[ 1033.634645]  __dev_queue_xmit+0x30e/0xa40
[ 1033.637014]  libiscsi_tcp
[ 1033.641281]  ? nf_route_table_hook4+0x8e/0x120 [nf_tables]
[ 1033.643458]  libiscsi
[ 1033.649017]  ip_finish_output2+0x269/0x430
[ 1033.651632]  scsi_transport_iscsi
[ 1033.656070]  ip_output+0x70/0xe0
[ 1033.658601]  ipmi_devintf ipmi_msghandler
[ 1033.663915]  ? __ip_finish_output+0x1c0/0x1c0
[ 1033.666448]  fuse
[ 1033.673159]  __ip_queue_xmit+0x15d/0x430
[ 1033.676476] 
[ 1033.680669]  __tcp_transmit_skb+0x9fd/0xb50
[ 1033.683380] Red Hat flags:
[ 1033.688009]  ? __alloc_skb+0x82/0x1c0
[ 1033.690985]  eBPF/cgroup
[ 1033.697096]  ? tcp_delack_timer_handler+0x170/0x170
[ 1033.699721] 
[ 1033.704440]  tcp_delack_timer_handler+0x9e/0x170
[ 1033.708114] CPU: 38 PID: 0 Comm: swapper/38 Tainted: G          I  L   --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[ 1033.711962]  tcp_delack_timer+0x8a/0xa0
[ 1033.716327] Hardware name: Dell Inc. PowerEdge R640/0X45NX, BIOS 2.8.2 08/27/2020
[ 1033.721299]  call_timer_fn+0x2d/0x130
[ 1033.723578] RIP: 0010:native_queued_spin_lock_slowpath+0x16f/0x1b0
[ 1033.728116]  run_timer_softirq+0x1d8/0x410
[ 1033.729963] Code: 02 48 85 c0 74 1e 48 89 c1 0f 0d 08 eb 1c f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 04 d6 ad 61 c3 31 c9 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01
[ 1033.734765]  ? tick_sched_timer+0x37/0x70
[ 1033.737830] RSP: 0018:ffffb4818d2d4e88 EFLAGS: 00000202
[ 1033.742118]  __do_softirq+0xd7/0x2c4
[ 1033.745002]  ORIG_RAX: ffffffffffffff13
[ 1033.750501]  irq_exit_rcu+0xcb/0xd0
[ 1033.752343] RAX: 00000000009c0101 RBX: ffff981cfba156c0 RCX: 0000000000000000
[ 1033.757587]  irq_exit+0xa/0x10
[ 1033.768562] RDX: ffff983260eebb80 RSI: 00000000009c0000 RDI: ffff981cfba15288
[ 1033.773056]  smp_apic_timer_interrupt+0x74/0x130
[ 1033.780927] RBP: ffff981cfba15288 R08: ffff983260edaaa8 R09: ffffb4818d2d4ef8
[ 1033.785269]  apic_timer_interrupt+0xf/0x20
[ 1033.791849] R10: 0002ce08212b0a18 R11: 000000e3c6082a00 R12: ffff981cfba15200
[ 1033.796652]  </IRQ>
[ 1033.816259] R13: ffffffff9ec9e630 R14: ffff981cfba156c0 R15: 0000000000000002
[ 1033.821034] RIP: 0010:cpuidle_enter_state+0xda/0x3d0
[ 1033.826696] FS:  0000000000000000(0000) GS:ffff983260ec0000(0000) knlGS:0000000000000000
[ 1033.831043] Code: e8 2b 69 9c ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 a6 02 00 00 31 ff e8 7d 3f a3 ff fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 d6 48 8b 4c 24 10 48 2b 0c 24 48
[ 1033.835320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1033.839578] RSP: 0018:ffffb4818c9e7e58 EFLAGS: 00000246
[ 1033.847168] CR2: 00007f6901375a38 CR3: 00000024f9410004 CR4: 00000000007706e0
[ 1033.851010]  ORIG_RAX: ffffffffffffff13
[ 1033.858610] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1033.864050] RAX: ffff983260d6ae40 RBX: ffffffffa00b8528 RCX: 000000000000001f
[ 1033.871658] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1033.876593] RDX: 000000e3c0fe090c RSI: 0000000037c86db9 RDI: 0000000000000000
[ 1033.884208] PKRU: 55555554
[ 1033.887160] RBP: ffffd46980d40210 R08: 0000000000000002 R09: 000000000002a680
[ 1033.894800] Call Trace:
[ 1033.900646] R10: 0002ce080a76d88c R11: ffff983260d69b44 R12: 0000000000000003
[ 1033.909253]  <IRQ>
[ 1033.929879] R13: ffffffffa00b83c0 R14: 0000000000000003 R15: 0000000000000003
[ 1033.936173]  _raw_spin_lock+0x1a/0x20
[ 1033.942356]  ? cpuidle_enter_state+0xb5/0x3d0
[ 1033.950055]  tcp_write_timer+0x22/0x80
[ 1033.954869]  cpuidle_enter+0x2c/0x40
[ 1033.962564]  call_timer_fn+0x2d/0x130
[ 1033.970687]  do_idle+0x264/0x2c0
[ 1033.978387]  run_timer_softirq+0x1d8/0x410
[ 1033.986520]  cpu_startup_entry+0x6f/0x80
[ 1033.989800]  __do_softirq+0xd7/0x2c4
[ 1033.997959]  start_secondary+0x1a6/0x1e0
[ 1034.000992]  irq_exit_rcu+0xcb/0xd0
[ 1034.009170]  secondary_startup_64_no_verify+0xc2/0xcb
[ 1034.011779]  irq_exit+0xa/0x10
[ 1034.077091]  smp_apic_timer_interrupt+0x74/0x130
[ 1034.082264]  apic_timer_interrupt+0xf/0x20
[ 1034.086902]  </IRQ>
[ 1034.089530] RIP: 0010:cpuidle_enter_state+0xda/0x3d0
[ 1034.095024] Code: e8 2b 69 9c ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 a6 02 00 00 31 ff e8 7d 3f a3 ff fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 d6 48 8b 4c 24 10 48 2b 0c 24 48
[ 1034.114881] RSP: 0018:ffffb4818ca47e58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1034.123016] RAX: ffff983260eeae40 RBX: ffffffffa00b8528 RCX: 000000000000001f
[ 1034.130718] RDX: 000000e3cae2febc RSI: 0000000037c86db9 RDI: 0000000000000000
[ 1034.138408] RBP: ffffd46980ec0210 R08: 0000000000000002 R09: 000000000002a680
[ 1034.146101] R10: 0002ce08212aff80 R11: ffff983260ee9b44 R12: 0000000000000003
[ 1034.153789] R13: ffffffffa00b83c0 R14: 0000000000000003 R15: 0000000000000003
[ 1034.161470]  ? cpuidle_enter_state+0xb5/0x3d0
[ 1034.166376]  cpuidle_enter+0x2c/0x40
[ 1034.170494]  do_idle+0x264/0x2c0
[ 1034.174256]  cpu_startup_entry+0x6f/0x80
[ 1034.178704]  start_secondary+0x1a6/0x1e0
[ 1034.183145]  secondary_startup_64_no_verify+0xc2/0xcb
[ 1038.193754] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1038.200255] rcu: 	26-....: (5 GPs behind) idle=07a/1/0x4000000000000002 softirq=18452/18452 fqs=13154 
[ 1038.210114] 	(detected by 57, t=60018 jiffies, g=370137, q=181000)
[ 1038.216807] Sending NMI from CPU 57 to CPUs 26:
[ 1038.222086] NMI backtrace for cpu 26
[ 1038.222088] CPU: 26 PID: 0 Comm: swapper/26 Tainted: G          I  L   --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[ 1038.222090] Hardware name: Dell Inc. PowerEdge R640/0X45NX, BIOS 2.8.2 08/27/2020
[ 1038.222091] RIP: 0010:native_queued_spin_lock_slowpath+0x5d/0x1b0
[ 1038.222092] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 47 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00 00 75
[ 1038.222093] RSP: 0018:ffffb4818d063e18 EFLAGS: 00000202
[ 1038.222095] RAX: 00000000009c0101 RBX: ffff981bd1376900 RCX: 0000000000000020
[ 1038.222096] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff981cfba15288
[ 1038.222097] RBP: ffff981cfba15200 R08: 0000000000000001 R09: 0000000000000000
[ 1038.222098] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1038.222099] R13: ffff981cfba15288 R14: ffff9833018728f0 R15: ffff9833018728dc
[ 1038.222100] FS:  0000000000000000(0000) GS:ffff983260d40000(0000) knlGS:0000000000000000
[ 1038.222101] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1038.222101] CR2: 00007f73c2d34948 CR3: 00000024f9410004 CR4: 00000000007706e0
[ 1038.222102] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1038.222103] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1038.222104] PKRU: 55555554
[ 1038.222105] Call Trace:
[ 1038.222105]  <IRQ>
[ 1038.222106]  _raw_spin_lock+0x1a/0x20
[ 1038.222106]  tcp_v4_rcv+0xad5/0xc50
[ 1038.222107]  ip_protocol_deliver_rcu+0x2c/0x1d0
[ 1038.222108]  ip_local_deliver_finish+0x4d/0x60
[ 1038.222108]  ip_local_deliver+0xe0/0xf0
[ 1038.222109]  ? ip_protocol_deliver_rcu+0x1d0/0x1d0
[ 1038.222109]  ip_rcv+0x27b/0x36f
[ 1038.222110]  ? inet_add_protocol.cold.1+0x1e/0x1e
[ 1038.222111]  __netif_receive_skb_core+0x5cd/0xcb0
[ 1038.222111]  ? skb_clone+0x4d/0xb0
[ 1038.222112]  ? kmem_cache_alloc+0x13f/0x280
[ 1038.222112]  netif_receive_skb_internal+0x3d/0xb0
[ 1038.222113]  ? __skb_clone+0x29/0x130
[ 1038.222113]  tcf_mirred_act+0x241/0x510 [act_mirred]
[ 1038.222114]  tcf_action_exec+0x77/0x120
[ 1038.222115]  fl_classify+0x1e6/0x1f0 [cls_flower]
[ 1038.222115]  ? nf_conntrack_tcp_packet+0x33e/0xb90 [nf_conntrack]
[ 1038.222116]  tcf_classify+0x79/0x270
[ 1038.222116]  __dev_queue_xmit+0x30e/0xa40
[ 1038.222117]  ? nf_route_table_hook4+0x8e/0x120 [nf_tables]
[ 1038.222118]  ip_finish_output2+0x269/0x430
[ 1038.222118]  ip_output+0x70/0xe0
[ 1038.222119]  ? __ip_finish_output+0x1c0/0x1c0
[ 1038.222119]  __ip_queue_xmit+0x15d/0x430
[ 1038.222120]  __tcp_transmit_skb+0x9fd/0xb50
[ 1038.222121]  tcp_write_xmit+0x43d/0x12c0
[ 1038.222121]  ? tcp_rcv_established+0x2c5/0x5c0
[ 1038.222122]  __tcp_push_pending_frames+0x32/0xf0
[ 1038.222122]  tcp_rcv_established+0x2fb/0x5c0
[ 1038.222123]  ? tcp_v4_inbound_md5_hash+0x67/0x140
[ 1038.222124]  tcp_v4_do_rcv+0x12a/0x1e0
[ 1038.222124]  tcp_v4_rcv+0xb43/0xc50
[ 1038.222125]  ip_protocol_deliver_rcu+0x2c/0x1d0
[ 1038.222125]  ip_local_deliver_finish+0x4d/0x60
[ 1038.222126]  ip_local_deliver+0xe0/0xf0
[ 1038.222126]  ? ip_protocol_deliver_rcu+0x1d0/0x1d0
[ 1038.222127]  ip_rcv+0x27b/0x36f
[ 1038.222128]  ? inet_add_protocol.cold.1+0x1e/0x1e
[ 1038.222128]  __netif_receive_skb_core+0x5cd/0xcb0
[ 1038.222129]  ? skb_clone+0x4d/0xb0
[ 1038.222130]  ? kmem_cache_alloc+0x13f/0x280
[ 1038.222130]  netif_receive_skb_internal+0x3d/0xb0
[ 1038.222131]  ? __skb_clone+0x29/0x130
[ 1038.222132]  tcf_mirred_act+0x241/0x510 [act_mirred]
[ 1038.222132]  tcf_action_exec+0x77/0x120
[ 1038.222133]  fl_classify+0x1e6/0x1f0 [cls_flower]
[ 1038.222134]  ? nf_conntrack_tcp_packet+0x33e/0xb90 [nf_conntrack]
[ 1038.222135]  tcf_classify+0x79/0x270
[ 1038.222135]  __dev_queue_xmit+0x30e/0xa40
[ 1038.222136]  ? nf_route_table_hook4+0x8e/0x120 [nf_tables]
[ 1038.222137]  ip_finish_output2+0x269/0x430
[ 1038.222137]  ip_output+0x70/0xe0
[ 1038.222138]  ? __ip_finish_output+0x1c0/0x1c0
[ 1038.222139]  __ip_queue_xmit+0x15d/0x430
[ 1038.222139]  __tcp_transmit_skb+0x9fd/0xb50
[ 1038.222140]  ? __alloc_skb+0x82/0x1c0
[ 1038.222141]  ? tcp_delack_timer_handler+0x170/0x170
[ 1038.222141]  tcp_delack_timer_handler+0x9e/0x170
[ 1038.222142]  tcp_delack_timer+0x8a/0xa0
[ 1038.222142]  call_timer_fn+0x2d/0x130
[ 1038.222143]  run_timer_softirq+0x1d8/0x410
[ 1038.222143]  ? tick_sched_timer+0x37/0x70
[ 1038.222144]  __do_softirq+0xd7/0x2c4
[ 1038.222145]  irq_exit_rcu+0xcb/0xd0
[ 1038.222145]  irq_exit+0xa/0x10
[ 1038.222146]  smp_apic_timer_interrupt+0x74/0x130
[ 1038.222146]  apic_timer_interrupt+0xf/0x20
[ 1038.222147]  </IRQ>
[ 1038.222147] RIP: 0010:cpuidle_enter_state+0xda/0x3d0
[ 1038.222148] Code: e8 2b 69 9c ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 a6 02 00 00 31 ff e8 7d 3f a3 ff fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 d6 48 8b 4c 24 10 48 2b 0c 24 48
[ 1038.222150] RSP: 0018:ffffb4818c9e7e58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 1038.222151] RAX: ffff983260d6ae40 RBX: ffffffffa00b8528 RCX: 000000000000001f
[ 1038.222152] RDX: 000000e3c0fe090c RSI: 0000000037c86db9 RDI: 0000000000000000
[ 1038.222153] RBP: ffffd46980d40210 R08: 0000000000000002 R09: 000000000002a680
[ 1038.222153] R10: 0002ce080a76d88c R11: ffff983260d69b44 R12: 0000000000000003
[ 1038.222154] R13: ffffffffa00b83c0 R14: 0000000000000003 R15: 0000000000000003
[ 1038.222155]  ? cpuidle_enter_state+0xb5/0x3d0
[ 1038.222155]  cpuidle_enter+0x2c/0x40
[ 1038.222156]  do_idle+0x264/0x2c0
[ 1038.222157]  cpu_startup_entry+0x6f/0x80
[ 1038.222157]  start_secondary+0x1a6/0x1e0
[ 1038.222158]  secondary_startup_64_no_verify+0xc2/0xcb
[ 1048.125300] mlx5_cmd_check: 1160 callbacks suppressed
[ 1048.125303] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.145479] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.160168] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.174920] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.189748] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.205085] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.220214] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.235424] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.250559] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1048.265709] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14732): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)

============================================================================================================================================================
Kernel panic for flow 12-a: Host Pod -> NodePort Service traffic (Host Backend - Same Node).
[ 1624.638173] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [swapper/4:0]
[ 1624.646331] Modules linked in: act_tunnel_key nf_conntrack_netlink act_csum act_pedit veth act_ct nf_flow_table xt_addrtype ipt_REJECT nf_reject_ipv4 xt_nat xt_CT vhost_net vhost tap tun xt_REDIRECT ip6t_MASQUERADE ipt_MASQUERADE xt_conntrack xt_comment nft_counter xt_mark nft_compat nft_chain_nat nf_tables act_skbedit act_mirred geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink act_gact cls_flower sch_ingress openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mlx5_vdpa vringh vhost_iotlb vdpa overlay ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp kvm_intel mgag200 dell_smbios iTCO_wdt iTCO_vendor_support rpcrdma drm_kms_helper wmi_bmof dell_wmi_descriptor kvm dcdbas syscopyarea irqbypass sunrpc sysfillrect sysimgblt rapl fb_sys_fops intel_cstate acpi_ipmi mei_me pcspkr joydev i2c_i801 intel_uncore drm lpc_ich wmi mei ipmi_si acpi_power_meter ip_tables
[ 1624.646420]  xfs
[ 1624.667173] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [swapper/8:0]
[ 1624.744072]  libcrc32c
[ 1624.746783] Modules linked in:
[ 1624.755096]  rdma_ucm
[ 1624.758321]  act_tunnel_key
[ 1624.762706]  ib_srpt
[ 1624.765831]  nf_conntrack_netlink
[ 1624.769939]  ib_isert
[ 1624.772963]  act_csum
[ 1624.777575]  iscsi_target_mod
[ 1624.780676]  act_pedit
[ 1624.784235]  target_core_mod
[ 1624.788031]  veth act_ct
[ 1624.791684]  ib_iser
[ 1624.795386]  nf_flow_table
[ 1624.799204]  ib_umad
[ 1624.802204]  xt_addrtype
[ 1624.806169]  ib_ipoib
[ 1624.809159]  ipt_REJECT
[ 1624.812935]  rdma_cm
[ 1624.815984]  nf_reject_ipv4
[ 1624.819611]  iw_cm
[ 1624.822542]  xt_nat
[ 1624.826459]  ib_cm
[ 1624.829175]  xt_CT
[ 1624.832332]  mlx5_ib
[ 1624.834998]  vhost_net
[ 1624.838002]  ib_uverbs
[ 1624.840809]  vhost
[ 1624.844101]  ib_core
[ 1624.847032]  tap
[ 1624.849909]  sd_mod
[ 1624.852635]  tun
[ 1624.855272]  t10_pi
[ 1624.857860]  xt_REDIRECT
[ 1624.860432]  sg
[ 1624.862983]  ip6t_MASQUERADE
[ 1624.866185]  crct10dif_pclmul
[ 1624.868354]  ipt_MASQUERADE
[ 1624.871844]  mlx5_core
[ 1624.875197]  xt_conntrack
[ 1624.878551]  crc32_pclmul
[ 1624.881255]  xt_comment nft_counter
[ 1624.884373]  crc32c_intel
[ 1624.887299]  xt_mark
[ 1624.891235]  ahci
[ 1624.894126]  nft_compat
[ 1624.896690]  libahci
[ 1624.898840]  nft_chain_nat
[ 1624.901618]  psample
[ 1624.904006]  nf_tables act_skbedit
[ 1624.907015]  igb
[ 1624.909395]  act_mirred geneve
[ 1624.913090]  ixgbe
[ 1624.915126]  ip6_udp_tunnel
[ 1624.918484]  mlxfw
[ 1624.920692]  udp_tunnel nfnetlink_cttimeout
[ 1624.923782]  libata
[ 1624.925986]  nfnetlink act_gact
[ 1624.930475]  megaraid_sas
[ 1624.932776]  cls_flower
[ 1624.936221]  i2c_algo_bit
[ 1624.939043]  sch_ingress openvswitch
[ 1624.941794]  ghash_clmulni_intel
[ 1624.944615]  nf_conncount
[ 1624.948502]  pci_hyperv_intf
[ 1624.951940]  nf_nat
[ 1624.954871]  mdio
[ 1624.957962]  nf_conntrack
[ 1624.960380]  dca
[ 1624.962509]  nf_defrag_ipv6 nf_defrag_ipv4
[ 1624.965446]  dm_multipath
[ 1624.967494]  mlx5_vdpa
[ 1624.971914]  dm_mirror
[ 1624.974748]  vringh
[ 1624.977437]  dm_region_hash
[ 1624.980010]  vhost_iotlb
[ 1624.982442]  dm_log
[ 1624.985455]  vdpa overlay
[ 1624.988319]  dm_mod
[ 1624.990637]  ext4
[ 1624.993592]  be2iscsi
[ 1624.995909]  mbcache
[ 1624.998178]  bnx2i
[ 1625.000668]  jbd2
[ 1625.003192]  cnic
[ 1625.005424]  intel_rapl_msr
[ 1625.007679]  uio
[ 1625.009822]  intel_rapl_common
[ 1625.012945]  cxgb4i
[ 1625.014995]  isst_if_common skx_edac
[ 1625.018384]  cxgb4
[ 1625.020700]  nfit libnvdimm
[ 1625.024606]  tls(X)
[ 1625.026837]  x86_pkg_temp_thermal
[ 1625.029966]  libcxgbi
[ 1625.032285]  intel_powerclamp
[ 1625.035942]  libcxgb
[ 1625.038441]  ipmi_ssif
[ 1625.041749]  qla4xxx
[ 1625.044159]  coretemp kvm_intel
[ 1625.046859]  iscsi_boot_sysfs
[ 1625.049261]  mgag200
[ 1625.052742]  iscsi_tcp
[ 1625.055924]  dell_smbios
[ 1625.058442]  libiscsi_tcp
[ 1625.061017]  iTCO_wdt iTCO_vendor_support
[ 1625.063876]  libiscsi
[ 1625.066706]  rpcrdma drm_kms_helper
[ 1625.071041]  ipmi_devintf
[ 1625.073532]  wmi_bmof dell_wmi_descriptor
[ 1625.077356]  ipmi_msghandler
[ 1625.080192]  kvm
[ 1625.084536]  fuse
[ 1625.087639]  dcdbas
[ 1625.089818]  scsi_transport_iscsi
[ 1625.091958]  syscopyarea
[ 1625.094392] 
[ 1625.097925]  irqbypass
[ 1625.100802] Red Hat flags:
[ 1625.102513]  sunrpc sysfillrect
[ 1625.105201]  eBPF/cgroup
[ 1625.108121]  sysimgblt rapl
[ 1625.111596] 
[ 1625.114350]  fb_sys_fops
[ 1625.117479] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G        W I    X --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[ 1625.119191]  intel_cstate acpi_ipmi
[ 1625.122056] Hardware name: Dell Inc. PowerEdge R640/0X45NX, BIOS 2.8.2 08/27/2020
[ 1625.132723]  mei_me
[ 1625.136594] RIP: 0010:native_queued_spin_lock_slowpath+0x5b/0x1b0
[ 1625.144334]  pcspkr
[ 1625.146845] Code: 6d f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 47 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00
[ 1625.153217]  joydev i2c_i801
[ 1625.155753] RSP: 0018:ffffae880cbebe18 EFLAGS: 00000202
[ 1625.175139]  intel_uncore
[ 1625.178538]  ORIG_RAX: ffffffffffffff13
[ 1625.184099]  drm
[ 1625.187252] RAX: 0000000000240101 RBX: ffff98ae71355800 RCX: 0000000000000020
[ 1625.191429]  lpc_ich wmi
[ 1625.193798] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff98afe9146708
[ 1625.201288]  mei
[ 1625.204383] RBP: ffff98afe9146680 R08: 0000000000000001 R09: 0000000000000000
[ 1625.211888]  ipmi_si
[ 1625.214313] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1625.221835]  acpi_power_meter
[ 1625.224637] R13: ffff98afe9146708 R14: ffff98aea8dce4f0 R15: ffff98aea8dce4dc
[ 1625.232175]  ip_tables
[ 1625.235790] FS:  0000000000000000(0000) GS:ffff98c520a80000(0000) knlGS:0000000000000000
[ 1625.243352]  xfs libcrc32c
[ 1625.246385] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1625.254916]  rdma_ucm
[ 1625.258331] CR2: 00007f5435e5e948 CR3: 0000001b73810001 CR4: 00000000007706e0
[ 1625.264544]  ib_srpt
[ 1625.267550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1625.275161]  ib_isert iscsi_target_mod
[ 1625.278103] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1625.285728]  target_core_mod
[ 1625.290257] PKRU: 55555554
[ 1625.297904]  ib_iser
[ 1625.301585] Call Trace:
[ 1625.304808]  ib_umad
[ 1625.307787]  <IRQ>
[ 1625.310746]  ib_ipoib rdma_cm
[ 1625.313735]  _raw_spin_lock+0x1a/0x20
[ 1625.316259]  iw_cm
[ 1625.320030]  tcp_v4_rcv+0xad5/0xc50
[ 1625.324212]  ib_cm mlx5_ib
[ 1625.327031]  ip_protocol_deliver_rcu+0x2c/0x1d0
[ 1625.331032]  ib_uverbs
[ 1625.334548]  ip_local_deliver_finish+0x4d/0x60
[ 1625.339601]  ib_core sd_mod
[ 1625.342777]  ip_local_deliver+0xe0/0xf0
[ 1625.347745]  t10_pi
[ 1625.351372]  ? ip_protocol_deliver_rcu+0x1d0/0x1d0
[ 1625.355727]  sg
[ 1625.358621]  ip_rcv+0x27b/0x36f
[ 1625.363919]  crct10dif_pclmul
[ 1625.366447]  ? inet_add_protocol.cold.1+0x1e/0x1e
[ 1625.370076]  mlx5_core
[ 1625.373796]  __netif_receive_skb_core+0x5cd/0xcb0
[ 1625.378967]  crc32_pclmul
[ 1625.382038]  ? skb_clone+0x4d/0xb0
[ 1625.387191]  crc32c_intel
[ 1625.390513]  ? kmem_cache_alloc+0x13f/0x280
[ 1625.394361]  ahci
[ 1625.397670]  netif_receive_skb_internal+0x3d/0xb0
[ 1625.402299]  libahci
[ 1625.404904]  ? __skb_clone+0x29/0x130
[ 1625.410045]  psample
[ 1625.412902]  tcf_mirred_act+0x241/0x510 [act_mirred]
[ 1625.416978]  igb
[ 1625.419811]  tcf_action_exec+0x77/0x120
[ 1625.425187]  ixgbe
[ 1625.427658]  fl_classify+0x1e6/0x1f0 [cls_flower]
[ 1625.431886]  mlxfw
[ 1625.434508]  tcf_classify+0x79/0x270
[ 1625.439591]  libata
[ 1625.442187]  __dev_queue_xmit+0x30e/0xa40
[ 1625.446133]  megaraid_sas
[ 1625.448797]  ip_finish_output2+0x269/0x430
[ 1625.453163]  i2c_algo_bit
[ 1625.456331]  ip_output+0x70/0xe0
[ 1625.460775]  ghash_clmulni_intel
[ 1625.463918]  ? __ip_finish_output+0x1c0/0x1c0
[ 1625.467476]  pci_hyperv_intf
[ 1625.471212]  __ip_queue_xmit+0x15d/0x430
[ 1625.475888]  mdio
[ 1625.479264]  __tcp_transmit_skb+0x9fd/0xb50
[ 1625.483511]  dca
[ 1625.485945]  tcp_write_xmit+0x43d/0x12c0
[ 1625.490451]  dm_multipath
[ 1625.492809]  ? tcp_rcv_established+0x2c5/0x5c0
[ 1625.497067]  dm_mirror
[ 1625.500209]  __tcp_push_pending_frames+0x32/0xf0
[ 1625.504982]  dm_region_hash
[ 1625.507867]  tcp_rcv_established+0x2fb/0x5c0
[ 1625.512823]  dm_log
[ 1625.516148]  ? tcp_v4_inbound_md5_hash+0x67/0x140
[ 1625.520751]  dm_mod
[ 1625.523382]  tcp_v4_do_rcv+0x12a/0x1e0
[ 1625.528425]  be2iscsi
[ 1625.531057]  tcp_v4_rcv+0xb43/0xc50
[ 1625.535150]  bnx2i
[ 1625.537951]  ip_protocol_deliver_rcu+0x2c/0x1d0
[ 1625.541771]  cnic
[ 1625.544311]  ip_local_deliver_finish+0x4d/0x60
[ 1625.549181]  uio
[ 1625.551635]  ip_local_deliver+0xe0/0xf0
[ 1625.556418]  cxgb4i
[ 1625.558782]  ? ip_protocol_deliver_rcu+0x1d0/0x1d0
[ 1625.562955]  cxgb4
[ 1625.565578]  ip_rcv+0x27b/0x36f
[ 1625.570711]  tls(X)
[ 1625.573254]  ? inet_add_protocol.cold.1+0x1e/0x1e
[ 1625.576738]  libcxgbi
[ 1625.579365]  __netif_receive_skb_core+0x5cd/0xcb0
[ 1625.584410]  libcxgb qla4xxx
[ 1625.587214]  ? skb_clone+0x4d/0xb0
[ 1625.592261]  iscsi_boot_sysfs iscsi_tcp
[ 1625.595676]  ? kmem_cache_alloc+0x13f/0x280
[ 1625.599419]  libiscsi_tcp
[ 1625.603788]  netif_receive_skb_internal+0x3d/0xb0
[ 1625.608315]  libiscsi
[ 1625.611459]  ? __skb_clone+0x29/0x130
[ 1625.616504]  ipmi_devintf
[ 1625.619305]  tcf_mirred_act+0x241/0x510 [act_mirred]
[ 1625.623306]  ipmi_msghandler
[ 1625.626457]  tcf_action_exec+0x77/0x120
[ 1625.631757]  fuse
[ 1625.635174]  fl_classify+0x1e6/0x1f0 [cls_flower]
[ 1625.639351]  scsi_transport_iscsi
[ 1625.641814]  ? ip_local_deliver_finish+0x4d/0x60
[ 1625.646860] 
[ 1625.650722]  ? nf_conntrack_tcp_packet+0x33e/0xb90 [nf_conntrack]
[ 1625.655686] Red Hat flags:
[ 1625.657722]  tcf_classify+0x79/0x270
[ 1625.664170]  eBPF/cgroup
[ 1625.667437]  __dev_queue_xmit+0x30e/0xa40
[ 1625.671371] 
[ 1625.671372] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G        W I    X --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[ 1625.674466]  ? nf_route_table_hook4+0x8e/0x120 [nf_tables]
[ 1625.678835] Hardware name: Dell Inc. PowerEdge R640/0X45NX, BIOS 2.8.2 08/27/2020
[ 1625.680881]  ip_finish_output2+0x269/0x430
[ 1625.691682] RIP: 0010:native_queued_spin_lock_slowpath+0x16f/0x1b0
[ 1625.697757]  ip_output+0x70/0xe0
[ 1625.705622] Code: 02 48 85 c0 74 1e 48 89 c1 0f 0d 08 eb 1c f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 04 d6 6d 4d c3 31 c9 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01
[ 1625.710328]  ? __ip_finish_output+0x1c0/0x1c0
[ 1625.716905] RSP: 0018:ffffae880ccbce88 EFLAGS: 00000202
[ 1625.720754]  __ip_queue_xmit+0x15d/0x430
[ 1625.740359]  ORIG_RAX: ffffffffffffff13
[ 1625.745398]  __tcp_transmit_skb+0x9fd/0xb50
[ 1625.751059] RAX: 0000000000240101 RBX: ffff98afe9146b40 RCX: 0000000000000000
[ 1625.755672]  ? __alloc_skb+0x82/0x1c0
[ 1625.759952] RDX: ffff98c520b2bb80 RSI: 0000000000240000 RDI: ffff98afe9146708
[ 1625.764823]  ? tcp_delack_timer_handler+0x170/0x170
[ 1625.772403] RBP: ffff98afe9146708 R08: ffff98c520b1aaa8 R09: ffffae880ccbcef8
[ 1625.776772]  tcp_delack_timer_handler+0x9e/0x170
[ 1625.784367] R10: 0002d26cb70ff38c R11: 0000000000000000 R12: ffff98afe9146680
[ 1625.789979]  tcp_delack_timer+0x8a/0xa0
[ 1625.797588] R13: ffffffffb309e630 R14: ffff98afe9146b40 R15: 0000000000000082
[ 1625.802952]  call_timer_fn+0x2d/0x130
[ 1625.810577] FS:  0000000000000000(0000) GS:ffff98c520b00000(0000) knlGS:0000000000000000
[ 1625.815185]  run_timer_softirq+0x1d8/0x410
[ 1625.822816] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1625.827266]  ? tick_sched_timer+0x37/0x70
[ 1625.835866] CR2: 00007f7898000010 CR3: 0000001b73810004 CR4: 00000000007706e0
[ 1625.840779]  __do_softirq+0xd7/0x2c4
[ 1625.847044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1625.851886]  irq_exit_rcu+0xcb/0xd0
[ 1625.859556] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1625.863977]  irq_exit+0xa/0x10
[ 1625.871660] PKRU: 55555554
[ 1625.876013]  smp_apic_timer_interrupt+0x74/0x130
[ 1625.883711] Call Trace:
[ 1625.887647]  apic_timer_interrupt+0xf/0x20
[ 1625.890914]  <IRQ>
[ 1625.896411]  </IRQ>
[ 1625.899419]  _raw_spin_lock+0x1a/0x20
[ 1625.904402] RIP: 0010:cpuidle_enter_state+0xda/0x3d0
[ 1625.906981]  tcp_write_timer+0x22/0x80
[ 1625.909949] Code: e8 2b 69 9c ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 a6 02 00 00 31 ff e8 7d 3f a3 ff fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 d6 48 8b 4c 24 10 48 2b 0c 24 48
[ 1625.914171]  call_timer_fn+0x2d/0x130
[ 1625.920012] RSP: 0018:ffffae880c937e58 EFLAGS: 00000246
[ 1625.924320]  run_timer_softirq+0x1d8/0x410
[ 1625.944898]  ORIG_RAX: ffffffffffffff13
[ 1625.949148]  ? tick_sched_timer+0x37/0x70
[ 1625.955308] RAX: ffff98c520aaae40 RBX: ffffffffb44b8528 RCX: 000000000000001f
[ 1625.959997]  __do_softirq+0xd7/0x2c4
[ 1625.964746] RDX: 000001742a830169 RSI: 0000000037c86db9 RDI: 0000000000000000
[ 1625.969329]  irq_exit_rcu+0xcb/0xd0
[ 1625.977363] RBP: ffffce7000a80210 R08: 0000000000000002 R09: 000000000002a680
[ 1625.981511]  irq_exit+0xa/0x10
[ 1625.989550] R10: 0002d26ca12df950 R11: ffff98c520aa9b44 R12: 0000000000000003
[ 1625.993611]  smp_apic_timer_interrupt+0x74/0x130
[ 1626.001641] R13: ffffffffb44b83c0 R14: 0000000000000003 R15: 0000000000000003
[ 1626.005253]  apic_timer_interrupt+0xf/0x20
[ 1626.013264]  ? cpuidle_enter_state+0xb5/0x3d0
[ 1626.018444]  </IRQ>
[ 1626.026470]  cpuidle_enter+0x2c/0x40
[ 1626.031137] RIP: 0010:cpuidle_enter_state+0xda/0x3d0
[ 1626.036397]  do_idle+0x264/0x2c0
[ 1626.039066] Code: e8 2b 69 9c ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 a6 02 00 00 31 ff e8 7d 3f a3 ff fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 d6 48 8b 4c 24 10 48 2b 0c 24 48
[ 1626.043518]  cpu_startup_entry+0x6f/0x80
[ 1626.049033] RSP: 0018:ffffae880c957e58 EFLAGS: 00000246
[ 1626.053121]  start_secondary+0x1a6/0x1e0
[ 1626.073021]  ORIG_RAX: ffffffffffffff13
[ 1626.077826]  secondary_startup_64_no_verify+0xc2/0xcb
[ 1626.083622] RAX: ffff98c520b2ae40 RBX: ffffffffb44b8528 RCX: 000000000000001f
[ 1626.106103] RDX: 00000174340c4f21 RSI: 0000000037c86db9 RDI: 0000000000000000
[ 1626.113787] RBP: ffffce7000b00210 R08: 0000000000000002 R09: 000000000002a680
[ 1626.121466] R10: 0002d26cb70fcfa4 R11: ffff98c520b29b44 R12: 0000000000000003
[ 1626.129136] R13: ffffffffb44b83c0 R14: 0000000000000003 R15: 0000000000000003
[ 1626.136803]  ? cpuidle_enter_state+0xb5/0x3d0
[ 1626.141696]  cpuidle_enter+0x2c/0x40
[ 1626.145804]  do_idle+0x264/0x2c0
[ 1626.149555]  cpu_startup_entry+0x6f/0x80
[ 1626.153987]  start_secondary+0x1a6/0x1e0
[ 1626.158425]  secondary_startup_64_no_verify+0xc2/0xcb
[ 1633.415982] mlx5_cmd_check: 1160 callbacks suppressed
[ 1633.415985] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.436498] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.451435] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.466446] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.481502] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.497079] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.512402] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.527773] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.543034] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)
[ 1633.558375] mlx5_core 0000:3b:00.0: mlx5_cmd_check:780:(pid 14986): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x1e5585)

OVS: openvswitch2.17-2.17.0-22.bz2099846.5.2.el8fdp.x86_64
Kernel: Version comes with OCP 4.11 coreos, running 8.6: Linux worker-advnetlab49 4.18.0-372.19.1.el8_6.x86_64 #1 SMP Mon Jul 18 11:14:02 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Only OVS was upgraded with the patch. No other parts of the system was changed manually. Running the nightly OCP 4.11 release: 4.11.0-0.nightly-2022-08-26-123526

Comment 7 Balazs Nemeth 2022-12-28 14:29:51 UTC

*** Bug 2098174 has been marked as a duplicate of this bug. ***

Comment 10 Marcelo Ricardo Leitner 2023-01-25 22:08:01 UTC

Hi Li. Can we have an ITM here please?

Comment 11 Marcelo Ricardo Leitner 2023-01-25 22:12:06 UTC

We need this fixed in 8.6 as it is blocking OCP in there, and 8.4.z because the fix in ovs (for OCP) will start triggering the issue in there as well.

Comment 14 Li Shuang 2023-01-28 02:25:29 UTC

(In reply to Marcelo Ricardo Leitner from comment #10)
> Hi Li. Can we have an ITM here please?
Sure.

Comment 26 errata-xmlrpc 2023-05-16 08:52:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2951

Comment 27 errata-xmlrpc 2023-05-16 09:13:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2951