Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1949887

Summary: [broadcom][ovs dpdk bonding] Both openvswitch-2.9.9-1.el7fdp.x86_64 and kernel crashed after create ovs dpdk boding
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Hekai Wang <hewang>
Component: openvswitchAssignee: Timothy Redaelli <tredaelli>
openvswitch sub component: daemons and tools QA Contact: Hekai Wang <hewang>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: ctrautma, jhsiao, qding
Version: FDP 19.C   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-09 09:01:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ovs log
none
console log none

Description Hekai Wang 2021-04-15 10:33:02 UTC 
Created attachment 1772110 [details]
ovs log

Description of problem:
ovs crashed after create ovs dpdk bonding with broadcom nic cards .
At meanwhile kernel core dump as well

Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00101|dpdk|ERR|PMD: bnxt_hwrm_vnic_rss_cfg error 2:0:00000000:0310
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00102|dpdk|ERR|PMD: HWRM vnic 0 set RSS failure rc: ffffffea
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00103|dpdk|ERR|PMD: bnxt_hwrm_vnic_tpa_cfg error 2:0:00000000:0293
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00104|dpdk|EMER|PANIC in rte_free():
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-ctl[97389]: Starting ovs-vswitchd 2021-04-15T10:16:48Z|00104|dpdk|EMER|PANIC in rte_free():
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-ctl[97389]: 2021-04-15T10:16:48Z|00105|dpdk|EMER|Fatal error: Invalid memory
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00105|dpdk|EMER|Fatal error: Invalid memory
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00106|dpdk|ERR|19: [ovs-vswitchd(+0x40cd5) [0x5644fcad9cd5]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00107|dpdk|ERR|18: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2f8c108555]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00108|dpdk|ERR|17: [ovs-vswitchd(+0x3fedd) [0x5644fcad8edd]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00109|dpdk|ERR|16: [ovs-vswitchd(+0x1ff839) [0x5644fcc98839]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00110|dpdk|ERR|15: [ovs-vswitchd(+0x1fc2e5) [0x5644fcc952e5]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00111|dpdk|ERR|14: [ovs-vswitchd(+0x1fa6b5) [0x5644fcc936b5]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00112|dpdk|ERR|13: [ovs-vswitchd(+0x20c991) [0x5644fcca5991]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00113|dpdk|ERR|12: [ovs-vswitchd(+0x216180) [0x5644fccaf180]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00114|dpdk|ERR|11: [ovs-vswitchd(+0x2659ae) [0x5644fccfe9ae]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00115|dpdk|ERR|10: [ovs-vswitchd(+0x25f3ed) [0x5644fccf83ed]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00116|dpdk|ERR|9: [ovs-vswitchd(+0x25f267) [0x5644fccf8267]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00117|dpdk|ERR|8: [ovs-vswitchd(+0x25e6e9) [0x5644fccf76e9]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00118|dpdk|ERR|7: [ovs-vswitchd(+0x33cccc) [0x5644fcdd5ccc]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00119|dpdk|ERR|6: [ovs-vswitchd(+0x63b35) [0x5644fcafcb35]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00120|dpdk|ERR|5: [ovs-vswitchd(+0x93764) [0x5644fcb2c764]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00121|dpdk|ERR|4: [ovs-vswitchd(+0xa0f78) [0x5644fcb39f78]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00122|dpdk|ERR|3: [ovs-vswitchd(+0x58300) [0x5644fcaf1300]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00123|dpdk|ERR|2: [ovs-vswitchd(+0x3581d) [0x5644fcace81d]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97415]: ovs|00124|dpdk|ERR|1: [ovs-vswitchd(+0x5099d) [0x5644fcae999d]]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97414]: ovs|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-vswitchd[97414]: ovs|00003|daemon_unix|EMER|could not detach from foreground session
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-ctl[97389]: ovs-vswitchd: could not detach from foreground session
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com ovs-ctl[97389]: [FAILED]
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com systemd[1]: ovs-vswitchd.service: control process exited, code=exited status=1
Apr 15 06:16:48 dell-per740-10.rhts.eng.pek2.redhat.com systemd[1]: Failed to start Open vSwitch Forwarding Unit.

Here is the kernel call trace 
[  140.149870] ------------[ cut here ]------------ 
[  140.154501] WARNING: CPU: 26 PID: 5138 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x64/0x80 
[  140.162239] sysfs: cannot create duplicate filename '/class/net/bonding_masters' 
[  140.169619] Modules linked in: bonding openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 sctp mlx4_ib mlx4_en mlx4_core xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt i40iw rpcrdma sunrpc rdma_ucm ib_iser rdma_cm iw_cm libiscsi scsi_transport_iscsi ib_umad ib_ipoib ib_cm mlx5_ib ib_uverbs dell_smbios dell_wmi_descriptor iTCO_wdt iTCO_vendor_support dcdbas skx_edac bnxt_re intel_powerclamp ib_core coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ipmi_ssif cryptd pcspkr sg wmi ipmi_si ipmi_devintf ipmi_msghandler mei_me mei i2c_i801 lpc_ich acpi_power_meter acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops libahci ttm mlx5_core i40e tg3 crct10dif_pclmul crct10dif_common mlxfw drm bnxt_en crc32c_intel libata ptp devlink megaraid_sas pps_core drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod 
[  140.294971] CPU: 26 PID: 5138 Comm: ovs-dpctl Kdump: loaded Not tainted 3.10.0-1160.24.1.el7.x86_64 #1 
[  140.304246] Hardware name: Dell Inc. PowerEdge R740/0JMK61, BIOS 2.10.2 02/24/2021 
[  140.311796] Call Trace: 
[  140.314251]  [<ffffffffbc38308a>] dump_stack+0x19/0x1b 
[  140.319381]  [<ffffffffbbc9b1b8>] __warn+0xd8/0x100 
[  140.324254]  [<ffffffffbbc9b23f>] warn_slowpath_fmt+0x5f/0x80 
[  140.329987]  [<ffffffffbbed9338>] ? kernfs_path+0x48/0x60 
[  140.335372]  [<ffffffffbbedc044>] sysfs_warn_dup+0x64/0x80 
[  140.340846]  [<ffffffffbbedc404>] sysfs_do_create_link_sd.isra.2+0xb4/0xc0 
[  140.347702]  [<ffffffffbbedc435>] sysfs_create_link+0x25/0x50 
[  140.353435]  [<ffffffffbc0b7f77>] device_add+0x397/0x7c0 
[  140.358735]  [<ffffffffbc0b5e67>] ? dev_set_name+0x57/0x80 
[  140.364208]  [<ffffffffbc277f16>] netdev_register_kobject+0x96/0x190 
[  140.370546]  [<ffffffffbbccbf86>] ? raw_notifier_call_chain+0x16/0x20 
[  140.376980]  [<ffffffffbc25c204>] register_netdevice+0x524/0x770 
[  140.382978]  [<ffffffffc0d10766>] internal_dev_create+0x106/0x1a0 [openvswitch] 
[  140.390264]  [<ffffffffc0d0fd2c>] ovs_vport_add+0xdc/0x140 [openvswitch] 
[  140.396948]  [<ffffffffc0d02cf2>] new_vport+0x12/0x50 [openvswitch] 
[  140.403201]  [<ffffffffc0d05eb3>] ovs_dp_cmd_new+0x1a3/0x360 [openvswitch] 
[  140.410055]  [<ffffffffbc295ea8>] genl_family_rcv_msg+0x208/0x430 
[  140.416131]  [<ffffffffbc29082f>] ? __netlink_sendskb+0x5f/0x180 
[  140.422125]  [<ffffffffbbf07bac>] ? security_sock_rcv_skb+0x1c/0x20 
[  140.428373]  [<ffffffffbc29612b>] genl_rcv_msg+0x5b/0xc0 
[  140.433674]  [<ffffffffbc2960d0>] ? genl_family_rcv_msg+0x430/0x430 
[  140.439925]  [<ffffffffbc29411b>] netlink_rcv_skb+0xab/0xc0 
[  140.445482]  [<ffffffffbc294658>] genl_rcv+0x28/0x40 
[  140.450438]  [<ffffffffbc293aa0>] netlink_unicast+0x170/0x210 
[  140.456168]  [<ffffffffbc293e48>] netlink_sendmsg+0x308/0x420 
[  140.461903]  [<ffffffffbc2363a6>] sock_sendmsg+0xb6/0xf0 
[  140.467203]  [<ffffffffbbf8d622>] ? radix_tree_lookup_slot+0x22/0x50 
[  140.473542]  [<ffffffffbbdc0c8d>] ? filemap_fault+0x17d/0x420 
[  140.479279]  [<ffffffffbc237269>] ___sys_sendmsg+0x3e9/0x400 
[  140.484924]  [<ffffffffbc292198>] ? netlink_insert+0x1b8/0x340 
[  140.490746]  [<ffffffffbbf0c2a5>] ? sock_has_perm+0x75/0x90 
[  140.496358]  [<ffffffffbbe4a9aa>] ? __check_object_size+0x1ca/0x250 
[  140.502613]  [<ffffffffbc235e81>] ? move_addr_to_user+0xa1/0xe0 
[  140.508518]  [<ffffffffbc2361c2>] ? SYSC_getsockname+0xd2/0xf0 
[  140.514338]  [<ffffffffbc238921>] __sys_sendmsg+0x51/0x90 
[  140.519724]  [<ffffffffbc238972>] SyS_sendmsg+0x12/0x20 
[  140.524940]  [<ffffffffbc396226>] tracesys+0xa6/0xcc 
[  140.529890] ---[ end trace fb8683c8f97605f6 ]--- 



Version-Release number of selected component (if applicable):
[root@dell-per740-10 ~]# rpm -qa | grep openv
openvswitch-selinux-extra-policy-1.0-18.el7fdp.noarch
kernel-kernel-networking-openvswitch-common-2.0-122.noarch
openvswitch-2.9.9-1.el7fdp.x86_64
[root@dell-per740-10 ~]# rpm -qa | grep dpdk
kernel-kernel-networking-ovs-dpdk-bonding-new-bonding-1.0-208.noarch
dpdk-tools-18.11.8-1.el7_8.x86_64
dpdk-18.11.8-1.el7_8.x86_64
[root@dell-per740-10 ~]# uname  -a
Linux dell-per740-10.rhts.eng.pek2.redhat.com 3.10.0-1160.24.1.el7.x86_64 #1 SMP Thu Mar 25 21:21:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux


How reproducible:
Always

Steps to Reproduce:
Install hosts
enable dpdk with broadcom nic 
Create ovs dpdk bonding 

Job link https://beaker.engineering.redhat.com/jobs/5274038

Actual results:
Both kernel and ovs are crash and panic

Expected results:
It works fine .

Additional info:
Comment 1 Hekai Wang 2021-04-15 10:34:30 UTC 
Created attachment 1772111 [details]
console log