RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2020532 - ib_cache_gid_add: unable to add gid
Summary: ib_cache_gid_add: unable to add gid
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kernel
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Selvin Xavier (Broadcom)
QA Contact: Afom T. Michael
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-05 08:33 UTC by jnikolak
Modified: 2024-01-12 04:25 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-17 01:27:13 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-101821 0 None None None 2021-11-05 08:36:22 UTC

Description jnikolak 2021-11-05 08:33:33 UTC 
Description of problem:
We're seeing the following errors on one of our Supermicro "AS -2114GT-DNR" machines with Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [].



[1210135.703909] bnxt_en 0000:01:00.1: cmdq[0x33]=0x11 status 0x1
[1210135.708247] bnxt_en 0000:01:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[1210135.713444] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[1210135.717487] __ib_cache_gid_add: unable to add gid
Comment 2 Chen Chen 2022-01-26 05:13:48 UTC 
Also observed on 8.5 (4.18.0-348.12.2) with a BCM57414 card. Does it stop rdma from working?

[root@sirius-c03 ~]# lspci -v | grep -A30 BCM
17:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01)
        DeviceName: NIC1
        Subsystem: Broadcom Inc. and subsidiaries NetXtreme E-Series Advanced Dual-port 25Gb SFP28 Network Daughter Card
        Flags: bus master, fast devsel, latency 0, IRQ 50, NUMA node 0
        Memory at 96e10000 (64-bit, prefetchable) [size=64K]
        Memory at 96d00000 (64-bit, prefetchable) [size=1M]
        Memory at 96e22000 (64-bit, prefetchable) [size=8K]
        Expansion ROM at 97000000 [disabled] [size=256K]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [a0] MSI-X: Enable+ Count=74 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Device Serial Number e4-3d-1a-ff-fe-e7-30-00
        Capabilities: [150] Power Budgeting <?>
        Capabilities: [160] Virtual Channel
        Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
        Capabilities: [1b0] Latency Tolerance Reporting
        Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [230] Transaction Processing Hints
        Capabilities: [300] Secondary PCI Express
        Capabilities: [200] Precision Time Measurement
        Kernel driver in use: bnxt_en
        Kernel modules: bnxt_en

17:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01)
        DeviceName: NIC2
        Subsystem: Broadcom Inc. and subsidiaries NetXtreme E-Series Advanced Dual-port 25Gb SFP28 Network Daughter Card
        Flags: bus master, fast devsel, latency 0, IRQ 59, NUMA node 0
        Memory at 96e00000 (64-bit, prefetchable) [size=64K]
        Memory at 96c00000 (64-bit, prefetchable) [size=1M]
        Memory at 96e20000 (64-bit, prefetchable) [size=8K]
        Expansion ROM at 97040000 [disabled] [size=256K]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [a0] MSI-X: Enable+ Count=74 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Device Serial Number e4-3d-1a-ff-fe-e7-30-00
        Capabilities: [150] Power Budgeting <?>
        Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
        Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [230] Transaction Processing Hints
        Kernel driver in use: bnxt_en
        Kernel modules: bnxt_en



some dmesg messages:
[  +0.013488] infiniband bnxt_re0: Device registered successfully
[  +0.000575] bnxt_en 0000:17:00.1: cmdq[0x5]=0x11 status 0x1
[  +0.000003] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000002] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=0
[  +0.000004] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
[  +0.000090] bnxt_en 0000:17:00.1: cmdq[0x6]=0x11 status 0x1
[  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000001] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=0
[  +0.000002] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
[  +0.000034] bnxt_en 0000:17:00.1: cmdq[0x7]=0x11 status 0x1
[  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000001] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[  +0.000002] __ib_cache_gid_add: unable to add gid 0000:0000:0000:0000:0000:ffff:c0a8:0a0b error=-14
[  +0.000048] bnxt_en 0000:17:00.1: cmdq[0x8]=0x11 status 0x1
[  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000001] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[  +0.000001] __ib_cache_gid_add: unable to add gid 0000:0000:0000:0000:0000:ffff:c0a8:0a0b error=-14
[  +0.000028] bnxt_en 0000:17:00.1: cmdq[0x9]=0x11 status 0x1
[  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000001] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[  +0.000001] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
[  +0.000031] bnxt_en 0000:17:00.1: cmdq[0xa]=0x11 status 0x1
[  +0.000000] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000001] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[  +0.000001] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
[  +0.060768] bnxt_en 0000:17:00.1: cmdq[0x11]=0x15 status 0x5
[  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to allocate HW AH for Shadow QP
[  +0.000038] bnxt_en 0000:17:00.1 bnxt_re1: Failed to create AH entry for ShadowQP
[  +0.000066] infiniband bnxt_re1: Couldn't create ib_mad QP1
[  +0.000110] infiniband bnxt_re1: Couldn't open port 1
[  +0.000368] infiniband bnxt_re1: Device registered successfully
Comment 3 selvin.xavier 2022-01-26 05:55:25 UTC 
All these errors are seen on the Second PCI function.
Can you provide the output of 

ifconfig -a  or  ip link
ibv_devices
Comment 4 Chen Chen 2022-01-27 04:50:35 UTC 
The mentioned netif is eth0 and eth1.

[root@sirius-c03 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:e7:30:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:e7:30:00 brd ff:ff:ff:ff:ff:ff permaddr e4:3d:1a:e7:30:01
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:97:77:e2 brd ff:ff:ff:ff:ff:ff
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:97:77:e3 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:e7:30:00 brd ff:ff:ff:ff:ff:ff


[root@sirius-c03 ~]# ibv_devices
    device                 node GUID
    ------              ----------------
    bnxt_re0            e63d1afffee73000
    bnxt_re1            e63d1afffee73000

So it seems 802.3ad bonding interfered with RoCE setup process? Should I switch to alb or tlb instead?
Comment 5 Chen Chen 2022-01-27 04:55:26 UTC 
On the other hand, Mellanox ConnectX-4 Lx was successfully recognized as a bonding device. So driver of Broadcom lacked support of bonding.

[root@sirius ~]# ip link | grep -A3 ens1
6: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 50:6b:4b:1b:fc:72 brd ff:ff:ff:ff:ff:ff
7: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 50:6b:4b:1b:fc:72 brd ff:ff:ff:ff:ff:ff permaddr 50:6b:4b:1b:fc:73
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 50:6b:4b:1b:fc:72 brd ff:ff:ff:ff:ff:ff

[root@sirius ~]# ibv_devices
    device                 node GUID
    ------              ----------------
    mlx5_bond_0         506b4b03001bfc72
Comment 6 selvin.xavier 2022-01-27 05:52:48 UTC 
(In reply to Chen Chen from comment #5)
> On the other hand, Mellanox ConnectX-4 Lx was successfully recognized as a
> bonding device. So driver of Broadcom lacked support of bonding.
> 
> [root@sirius ~]# ip link | grep -A3 ens1
> 6: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master
> bond0 state UP mode DEFAULT group default qlen 1000
>     link/ether 50:6b:4b:1b:fc:72 brd ff:ff:ff:ff:ff:ff
> 7: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master
> bond0 state UP mode DEFAULT group default qlen 1000
>     link/ether 50:6b:4b:1b:fc:72 brd ff:ff:ff:ff:ff:ff permaddr
> 50:6b:4b:1b:fc:73
> 8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP mode DEFAULT group default qlen 1000
>     link/ether 50:6b:4b:1b:fc:72 brd ff:ff:ff:ff:ff:ff
> 
> [root@sirius ~]# ibv_devices
>     device                 node GUID
>     ------              ----------------
>     mlx5_bond_0         506b4b03001bfc72

You are right. Broadcom doesn't support LAG for RoCE with the upstream driver. 
The errors that you see is because of the Bond created with 2 slaves. Since second interface has the same mac address, these gid addition errors are expected for second function.   Traffic through first bnxt_re0 will continue to work. bnxt_re1 will not work.
Comment 7 Chen Chen 2022-01-28 09:38:44 UTC 
I tried "balance-alb 6" and slave interfaces retained their mac address. Node GUID differed, but similar errors still emits.

dmesg:
[  +0.146736] infiniband bnxt_re0: Device registered successfully
[  +0.000458] bnxt_en 0000:17:00.1: cmdq[0x5]=0x11 status 0x1
[  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
[  +0.000002] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=0
[  +0.000002] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
[  +0.000002] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee6:e1e0 error=-14

[root@sirius-c03 ~]# ibv_devices
    device                 node GUID
    ------              ----------------
    bnxt_re0            e63d1afffee73000
    bnxt_re1            e63d1afffee73001

[root@sirius-c03 ~]# ip link
...
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:e7:30:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:e7:30:01 brd ff:ff:ff:ff:ff:ff
...
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:3d:1a:e7:30:00 brd ff:ff:ff:ff:ff:ff

[root@sirius-c03 ~]# cat /sys/class/net/bond0/bonding/mode
balance-alb 6

"balance-rr 0" also tested and failed.
Comment 11 Curtis Taylor 2022-12-03 18:44:56 UTC 
selvin,

Earlier you wrote:  

  Broadcom doesn't support LAG for RoCE with the upstream driver. 
  The errors that you see is because of the Bond created with 2 slaves. 
  Since second interface has the same mac address, these gid addition errors are 
  expected for second function.   Traffic through first bnxt_re0 will continue to 
  work. bnxt_re1 will not work.

Subsequently, Chen Chen tested using balance-alb, which leaves the bond ports with their original hw addr, yet still got

  [  +0.146736] infiniband bnxt_re0: Device registered successfully
  [  +0.000458] bnxt_en 0000:17:00.1: cmdq[0x5]=0x11 status 0x1
  [  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
  [  +0.000002] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=0
  [  +0.000002] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
  [  +0.000002] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:e63d:1aff:fee6:e1e0 error=-14

Do you still believe the errors are due to the same mac address being assigned twice? (Honest question, all I know balance-alb resulted in attempting to add both MACs twice and the second attempt resulted in the failure.) 
I checked broadcom bnxt_qplib_add_sgid() and bnxt_re_add_gid() and did not find any comments nor warnings of bonding link aggregation not being supported (also googled to no avail for such).
Is it documented anywhere you know of that bnxt_en RoCE does not support any aggregate mode bond?
Is bonding with LAG support for RoCE planned in bnxt_en? OOT? upstream?

Should this bz be closed WONTFIX?

Thank you,
=- Curtis Taylor
Comment 13 selvin.xavier 2022-12-05 05:01:31 UTC 
T(In reply to Curtis Taylor from comment #11)
> selvin,
> 
> Earlier you wrote:  
> 
>   Broadcom doesn't support LAG for RoCE with the upstream driver. 
>   The errors that you see is because of the Bond created with 2 slaves. 
>   Since second interface has the same mac address, these gid addition errors
> are 
>   expected for second function.   Traffic through first bnxt_re0 will
> continue to 
>   work. bnxt_re1 will not work.
> 
> Subsequently, Chen Chen tested using balance-alb, which leaves the bond
> ports with their original hw addr, yet still got
> 
>   [  +0.146736] infiniband bnxt_re0: Device registered successfully
>   [  +0.000458] bnxt_en 0000:17:00.1: cmdq[0x5]=0x11 status 0x1
>   [  +0.000001] bnxt_en 0000:17:00.1 bnxt_re1: Failed to add GID: 0xfffffff2
>   [  +0.000002] infiniband bnxt_re1: add_roce_gid GID add failed port=1
> index=0
>   [  +0.000002] __ib_cache_gid_add: unable to add gid
> fe80:0000:0000:0000:e63d:1aff:fee7:3000 error=-14
>   [  +0.000002] __ib_cache_gid_add: unable to add gid
> fe80:0000:0000:0000:e63d:1aff:fee6:e1e0 error=-14
 
> 
> Do you still believe the errors are due to the same mac address being
> assigned twice? (Honest question, all I know balance-alb resulted in
> attempting to add both MACs twice and the second attempt resulted in the
> failure.) 
Second function is reporting this error. So when both ports (functions) becomes the part of the Bond, first port mac address will be used for Bond. So current FW fails any default GID addition to second port.

> I checked broadcom bnxt_qplib_add_sgid() and bnxt_re_add_gid() and did not
> find any comments nor warnings of bonding link aggregation not being
> supported (also googled to no avail for such).
> Is it documented anywhere you know of that bnxt_en RoCE does not support any
> aggregate mode bond?
> Is bonding with LAG support for RoCE planned in bnxt_en? OOT? upstream?

To support RoCE bond, we need to destroy the base interfaces (bnxt_re0 and bnxt_re1) and then create a single base interface (say bnxt_re_bond0).  Broadcom's OOT driver is supporting this (with some workarounds).  Ideally some support from IB stack is required to handle it cleanly. There was some discussion in community to have generic solution for all vendors (Some discussion happened earlier but didnt conclude - https://www.spinics.net/lists/linux-rdma/msg89227.html).  So for now, we can conclude that this feature is not supported in bnxt_re upstream.

> 
> Should this bz be closed WONTFIX?
Yes, we can close it as WONTFIX
> 
> Thank you,
> =- Curtis Taylor
Comment 15 Bin Hu 2023-06-07 12:44:11 UTC 
Hi Selvin,

Two more questions: 
1. What is the potential risk that customer didn't change anything and stay as is?  Does the current TX timeout issues badly affect this NIC working and it will lead potential unpredictable problems for applications? 
2. Currently customer is using bnxt_en driver version 1.10.1 which is shipped inside RHEL 8.4. I see the latest driver is 1.10.2-224.0.157.0 [1]. Will upgrade the driver version help to resolve those TX timeout issue?

By the way, customer already upgraded Firmware version to latest:

~~~
ens7f0np0  bnxt_en  fw 224.0.158.0/pkg 224.1.102000
ens7f1np1  bnxt_en  fw 224.0.158.0/pkg 224.1.102000
~~~

Thanks again. I will also send you the full log offline.

[1] https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_29a72f850e104ea8a2c1725f3e
Comment 16 Michal Tesar 2023-06-07 12:54:18 UTC 
Hello Selvin

We have another customer hitting the bond and roce limitation.
He is experiencing these errors in the dmesg:

1. I understand the "Failed to add GID" issue, which could be a workaround by changing the bonding type from mode 4 to mode 1 (active-backup).
 But what I am not sure is, whether the exposed 3 errors are related with GID issue:

[  368.476401] bnxt_en 0000:37:00.0 ens7f0np0: Error (timeout: 2000015) msg {0x23 0x4d0} len:0
[  372.023987] bnxt_en 0000:37:00.0 ens7f0np0: Error (timeout: 2000015) msg {0xb4 0x4d1} len:0
[  375.574062] bnxt_en 0000:37:00.1 ens7f1np1: Error (timeout: 2000015) msg {0x23 0x4ce} len:0
[  379.109100] bnxt_en 0000:37:00.1 ens7f1np1: Error (timeout: 2000015) msg {0xb4 0x4cf} len:0

and these:

[  787.047172] bnxt_en 0000:37:00.0 ens7f0np0: hwrm req_type 0xa1 seq id 0x5d5 error 0x3
[  787.055104] bnxt_en 0000:37:00.0 ens7f0np0: hwrm_tunnel_dst_port_alloc failed. rc:-13
[  787.063005] bnxt_en 0000:37:00.0 ens7f0np0: UDP tunnel port sync failed for table 0: -13
[  787.072835] bnxt_en 0000:37:00.1 ens7f1np1: hwrm req_type 0xa1 seq id 0x5d4 error 0x3
[  787.080729] bnxt_en 0000:37:00.1 ens7f1np1: hwrm_tunnel_dst_port_alloc failed. rc:-13
[  787.088622] bnxt_en 0000:37:00.1 ens7f1np1: UDP tunnel port sync failed for table 0: -13


 $ grep -E "bnxt|bond" sos_commands/logs/journalctl_--no-pager_--catalog_--boot_-1 | head -4
 Jun 06 00:54:22 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout detected, starting reset task!
 Jun 06 00:54:32 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout detected, starting reset task!
 Jun 06 00:54:40 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout detected, starting reset task!
 Jun 06 00:54:49 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout detected, starting reset task!

The question is if all these errors are related to the limitation or not.

The card in question is:
37:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)
    Subsystem: Hewlett Packard Enterprise Ethernet 25Gb 2-port 631SFP28 Adapter [1590:020e]
    Physical Slot: 7
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 28
    NUMA node: 0
    Region 0: Memory at e2a10000 (64-bit, prefetchable) [size=64K]
    Region 2: Memory at e2900000 (64-bit, prefetchable) [size=1M]
    Region 4: Memory at e2a22000 (64-bit, prefetchable) [size=8K]
    Expansion ROM at e2d00000 [virtual] [disabled] [size=256K]
    Capabilities: [48] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [50] Vital Product Data
        Product Name: HPE Eth 10/25Gb 2p 631SFP28 Adptr
        Read-only fields:
            [PN] Part number: 817716-001
            [EC] Engineering changes: A-6234
            [MN] Manufacture ID: 1590
            [V0] Vendor specific: SU 25G/10G GEN3 x8 14W
            [V1] Vendor specific: 224.0.155.0
            [V3] Vendor specific: 224.0.158.0
            [V5] Vendor specific: 0A
            [V6] Vendor specific: 224.0.155.0
            [V7] Vendor specific: 631SFP28
            [VA] Vendor specific: 224.1.102000
            [SN] Serial number: VNE1330FK3
            [V2] Vendor specific: 5133
            [V4] Vendor specific: 5CBA2C1A4770
            [RV] Reserved: checksum good, 154 byte(s) reserved
        End
...

Another question is if the card could be configured in its bios not to provide ROCE at all.
The customer claims that he has multiple machines with the same card and only this one is experiencing the errors.

Thank you for your help Michal Tesar
Comment 17 selvin.xavier 2023-06-07 13:11:31 UTC 
(In reply to Bin Hu from comment #15)
> Hi Selvin,
> 
> Two more questions: 
> 1. What is the potential risk that customer didn't change anything and stay
> as is?  Does the current TX timeout issues badly affect this NIC working and
> it will lead potential unpredictable problems for applications? 
If the tx timeout is frequent, then we need to debug this. tx timeout can cause a momentary  removal of device and driver re-initalizes the device while handling tx timeout. getting a FW dump would help us to identify if there is any issue in the HW.

ethtool -w <ethx> data fw_core_file.core

> 2. Currently customer is using bnxt_en driver version 1.10.1 which is
> shipped inside RHEL 8.4. I see the latest driver is 1.10.2-224.0.157.0 [1].
> Will upgrade the driver version help to resolve those TX timeout issue?
> 
Updating to 1.10.2-224.0.157.0 will be good. It has more fixes than the inbox driver.

> By the way, customer already upgraded Firmware version to latest:
> 
> ~~~
> ens7f0np0  bnxt_en  fw 224.0.158.0/pkg 224.1.102000
> ens7f1np1  bnxt_en  fw 224.0.158.0/pkg 224.1.102000
> ~~~
> 
> Thanks again. I will also send you the full log offline.
> 
> [1]
> https://support.hpe.com/connect/s/
> softwaredetails?language=en_US&softwareId=MTX_29a72f850e104ea8a2c1725f3e
Comment 18 selvin.xavier 2023-06-07 13:18:10 UTC 
(In reply to Michal Tesar from comment #16)
> Hello Selvin
> 
> We have another customer hitting the bond and roce limitation.
> He is experiencing these errors in the dmesg:
> 
> 1. I understand the "Failed to add GID" issue, which could be a workaround
> by changing the bonding type from mode 4 to mode 1 (active-backup).
>  But what I am not sure is, whether the exposed 3 errors are related with
> GID issue:
> 
> [  368.476401] bnxt_en 0000:37:00.0 ens7f0np0: Error (timeout: 2000015) msg
> {0x23 0x4d0} len:0
> [  372.023987] bnxt_en 0000:37:00.0 ens7f0np0: Error (timeout: 2000015) msg
> {0xb4 0x4d1} len:0
> [  375.574062] bnxt_en 0000:37:00.1 ens7f1np1: Error (timeout: 2000015) msg
> {0x23 0x4ce} len:0
> [  379.109100] bnxt_en 0000:37:00.1 ens7f1np1: Error (timeout: 2000015) msg
> {0xb4 0x4cf} len:0
> 
> and these:
> 
> [  787.047172] bnxt_en 0000:37:00.0 ens7f0np0: hwrm req_type 0xa1 seq id
> 0x5d5 error 0x3
> [  787.055104] bnxt_en 0000:37:00.0 ens7f0np0: hwrm_tunnel_dst_port_alloc
> failed. rc:-13
> [  787.063005] bnxt_en 0000:37:00.0 ens7f0np0: UDP tunnel port sync failed
> for table 0: -13
> [  787.072835] bnxt_en 0000:37:00.1 ens7f1np1: hwrm req_type 0xa1 seq id
> 0x5d4 error 0x3
> [  787.080729] bnxt_en 0000:37:00.1 ens7f1np1: hwrm_tunnel_dst_port_alloc
> failed. rc:-13
> [  787.088622] bnxt_en 0000:37:00.1 ens7f1np1: UDP tunnel port sync failed
> for table 0: -13
> 
> 
>  $ grep -E "bnxt|bond"
> sos_commands/logs/journalctl_--no-pager_--catalog_--boot_-1 | head -4
>  Jun 06 00:54:22 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout
> detected, starting reset task!
>  Jun 06 00:54:32 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout
> detected, starting reset task!
>  Jun 06 00:54:40 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout
> detected, starting reset task!
>  Jun 06 00:54:49 xxxxxxxx kernel: bnxt_en 0000:37:00.0 ens7f0np0: TX timeout
> detected, starting reset task!
> 
> The question is if all these errors are related to the limitation or not.

These errors shouldn't come with even with the RoCE driver limitation we mentioned before. RoCE will not work with this limitation. But L2 traffic should work fine. As i mentioned before, its better to get a FW dump for identifying the issue.
> 
> The card in question is:
> 37:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414
> NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)
>     Subsystem: Hewlett Packard Enterprise Ethernet 25Gb 2-port 631SFP28
> Adapter [1590:020e]
>     Physical Slot: 7
>     Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>     Latency: 0
>     Interrupt: pin A routed to IRQ 28
>     NUMA node: 0
>     Region 0: Memory at e2a10000 (64-bit, prefetchable) [size=64K]
>     Region 2: Memory at e2900000 (64-bit, prefetchable) [size=1M]
>     Region 4: Memory at e2a22000 (64-bit, prefetchable) [size=8K]
>     Expansion ROM at e2d00000 [virtual] [disabled] [size=256K]
>     Capabilities: [48] Power Management version 3
>         Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>     Capabilities: [50] Vital Product Data
>         Product Name: HPE Eth 10/25Gb 2p 631SFP28 Adptr
>         Read-only fields:
>             [PN] Part number: 817716-001
>             [EC] Engineering changes: A-6234
>             [MN] Manufacture ID: 1590
>             [V0] Vendor specific: SU 25G/10G GEN3 x8 14W
>             [V1] Vendor specific: 224.0.155.0
>             [V3] Vendor specific: 224.0.158.0
>             [V5] Vendor specific: 0A
>             [V6] Vendor specific: 224.0.155.0
>             [V7] Vendor specific: 631SFP28
>             [VA] Vendor specific: 224.1.102000
>             [SN] Serial number: VNE1330FK3
>             [V2] Vendor specific: 5133
>             [V4] Vendor specific: 5CBA2C1A4770
>             [RV] Reserved: checksum good, 154 byte(s) reserved
>         End
> ...
> 
> Another question is if the card could be configured in its bios not to
> provide ROCE at all.
> The customer claims that he has multiple machines with the same card and
> only this one is experiencing the errors.
RoCE can be disabled on these adapters. The option is available in the boot BIOS of the Network. 
Select the Broadcom adapter after selecting the "system setup" during boot (using F2 or del)
Then disable RDMA.

we can also disable rdma using some of the broadcom tools like bnxtnvm. Please let us know if the customer is not able to disable RDMA from boot BIOS.
> 
> Thank you for your help Michal Tesar
Comment 20 Red Hat Bugzilla 2024-01-12 04:25:06 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.