1570393 – [Broadcom CCX 7.6 FEAT]libbnxt_re: Add SRQ support on Broadcom adapters

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1570393 - [Broadcom CCX 7.6 FEAT]libbnxt_re: Add SRQ support on Broadcom adapters

Summary: [Broadcom CCX 7.6 FEAT]libbnxt_re: Add SRQ support on Broadcom adapters

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	rdma-core
Sub Component:
Version:	7.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	7.7
Assignee:	Jarod Wilson
QA Contact:	Afom T. Michael
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1614004
TreeView+	depends on / blocked

Reported:	2018-04-22 14:42 UTC by Selvin Xavier (Broadcom)
Modified:	2019-08-06 12:46 UTC (History)
CC List:	8 users (show)
Fixed In Version:	rdma-core-22-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-08-06 12:46:08 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:2100	0	None	None	None	2019-08-06 12:46:29 UTC

Description Selvin Xavier (Broadcom) 2018-04-22 14:42:12 UTC

Description of problem:

SRQ support is not available in libbnxt_re. We are preparing the patches to be pushed upstream. This needs to be pulled in once it is accepted in upstream.


Steps to Reproduce:
1. Load bnxt_re drivers and install rdma-core with libbnxt_re
2. Run ibv_srq_ping_pong


Actual results:
 
The test fails now since the support is not available in the library.



Additional info:

Comment 3 Mike Stowell 2018-04-23 13:15:40 UTC

Thanks for the test info in BZ description - adding qa_ack+

Comment 4 Don Dutile (Red Hat) 2018-04-23 18:20:57 UTC

Selvin,
It cannot be pulled in until it is (minimally) in a for-next, or for-rc branch of rdma-core.
Please list the upstream commit-id's once it is pulled in.

Then Jarod can assess whether he can pull it into RHEL-7.6 rdma-core, or not.

Comment 5 Selvin Xavier (Broadcom) 2018-04-24 01:05:42 UTC

(In reply to Don Dutile from comment #4)
> Selvin,
> It cannot be pulled in until it is (minimally) in a for-next, or for-rc
> branch of rdma-core.
> Please list the upstream commit-id's once it is pulled in.
> 
> Then Jarod can assess whether he can pull it into RHEL-7.6 rdma-core, or not.

Don,
 This is not submitted yet. My colleague would be pushing the patches to upstream this week. Once it is accepted, i will update this BZ with the commit id.

Thanks,
Selvin

Comment 6 Selvin Xavier (Broadcom) 2018-10-04 05:00:39 UTC

Jarod,
 This commit is in rdma-core now. 

commit 602bb7ce1502c2e874eec6f1f677fd0539ceba01 - bnxt_re: Add shared receive queue support to broadcom's roce driver

Thanks,
Selvin

Comment 7 Don Dutile (Red Hat) 2018-10-04 22:46:49 UTC

(In reply to Selvin Xavier from comment #6)
> Jarod,
>  This commit is in rdma-core now. 
> 
> commit 602bb7ce1502c2e874eec6f1f677fd0539ceba01 - bnxt_re: Add shared
> receive queue support to broadcom's roce driver
> 
> Thanks,
> Selvin

... are you sure it isn't in RHEL-7.6's rdma-core already?

Comment 8 Selvin Xavier (Broadcom) 2018-10-15 12:33:55 UTC

(In reply to Don Dutile from comment #7)
> (In reply to Selvin Xavier from comment #6)
> > Jarod,
> >  This commit is in rdma-core now. 
> > 
> > commit 602bb7ce1502c2e874eec6f1f677fd0539ceba01 - bnxt_re: Add shared
> > receive queue support to broadcom's roce driver
> > 
> > Thanks,
> > Selvin
> 
> ... are you sure it isn't in RHEL-7.6's rdma-core already?


This was posted on last month Sept 17. See the posting message.

https://www.spinics.net/lists/linux-rdma/msg69055.html

Also, i have confirmed from the source rpms rdma-core-17.2-3.el7.src.rpm and rdma-core-19-3.el8.src.rpm that this is not yet available.

Comment 9 Jarod Wilson 2018-10-15 16:43:54 UTC

I think this is likely 7.7 material with a z-stream backport now, but I'm not 100% certain on that. We'll also need a RHEL8 bug for it to get backported there as well.

Comment 10 Don Dutile (Red Hat) 2018-10-15 18:00:47 UTC

(In reply to Jarod Wilson from comment #9)
> I think this is likely 7.7 material with a z-stream backport now, but I'm
> not 100% certain on that. We'll also need a RHEL8 bug for it to get
> backported there as well.

+1, *but*, a (7.6)z-stream backport would require QE to re-test bnxt-re, as that enables new functionality (srpt uses SRQ if available), thus, it's not a 'bug fix' for z-stream, but new functionality.

Comment 11 Jarod Wilson 2018-10-15 18:13:26 UTC

Oh, right, this is new feature enablement, not a straight bug-fix, so probably no go on the z-stream backport for 7.6.

Comment 12 Selvin Xavier (Broadcom) 2018-10-16 12:07:45 UTC

Created a BZ for 8.x
https://bugzilla.redhat.com/show_bug.cgi?id=1639692

Comment 14 Afom T. Michael 2019-05-29 22:21:58 UTC

Marking verified as our test has passed on 3.10.0-1048.el7.x86_64. Also, the ibv_srq_pingpong test of our libibverbs-utils has passed.

[root@rdma-dev-25 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 Beta (Maipo)
[root@rdma-dev-25 ~]$ uname -r
3.10.0-1048.el7.x86_64
[root@rdma-dev-25 ~]$ ip a s bnxt_roce
6: bnxt_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 00:0a:f7:ea:ce:a0 brd ff:ff:ff:ff:ff:ff
    inet 172.31.40.125/24 brd 172.31.40.255 scope global noprefixroute dynamic bnxt_roce
       valid_lft 1679sec preferred_lft 1679sec
    inet6 fe80::20a:f7ff:feea:cea0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@rdma-dev-25 ~]$ ibv_devinfo
hca_id:	bnxt_re0
	transport:			InfiniBand (0)
	fw_ver:				212.0.106.0
	node_guid:			020a:f7ff:feea:cea0
	sys_image_guid:			020a:f7ff:feea:cea0
	vendor_id:			0x14e4
	vendor_part_id:			5652
	hw_ver:				0x4540
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

[root@rdma-dev-25 ~]$ rpm -qa | grep -E 'rdma|verbs'
libibverbs-22.1-1.el7.x86_64
librdmacm-utils-22.1-1.el7.x86_64
rdma-core-22.1-1.el7.x86_64
libibverbs-utils-22.1-1.el7.x86_64
kernel-kernel-infiniband-librdmacm-utils-0.1-53.noarch
kernel-kernel-infiniband-libibverbs-utils-0.1-36.noarch
librdmacm-22.1-1.el7.x86_64
kernel-kernel-infiniband-nfsordma-1.0-11.noarch
rdma-core-devel-22.1-1.el7.x86_64
[root@rdma-dev-25 ~]$ rpm -ql libibverbs | grep libbnxt_re
/usr/lib64/libibverbs/libbnxt_re-rdmav22.so
[root@rdma-dev-25 ~]$ ibv_srq_pingpong -g 0 -d bnxt_re0 -i 1
  local address:  LID 0x0000, QPN 0x001022, PSN 0xa4ff12, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001023, PSN 0x40a3cb, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001024, PSN 0xcd2d83, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001025, PSN 0xb43edf, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001026, PSN 0xb7483d, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001027, PSN 0x4e7225, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001028, PSN 0x8325ea, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001029, PSN 0xee2965, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102a, PSN 0x0cc46f, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102b, PSN 0x8f678e, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102c, PSN 0x885450, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102d, PSN 0x3cf9da, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102e, PSN 0xfab888, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102f, PSN 0xc0385e, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001030, PSN 0x1a6bfa, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001031, PSN 0x7ad86c, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001022, PSN 0x414c12, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001023, PSN 0xfb6ccb, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001024, PSN 0xeac283, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001025, PSN 0x2aafdf, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001026, PSN 0xab653d, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001027, PSN 0x4ccb25, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001028, PSN 0xc50aea, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001029, PSN 0xbaaa65, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102a, PSN 0x05b16f, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102b, PSN 0x7a508e, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102c, PSN 0xe38950, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102d, PSN 0xb08ada, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102e, PSN 0x757588, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102f, PSN 0xd0b15e, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001030, PSN 0xd3f0fa, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001031, PSN 0xf6796c, GID fe80::20a:f7ff:feea:cd90
8192000 bytes in 0.00 seconds = 40604.71 Mbit/sec
1000 iters in 0.00 seconds = 1.61 usec/iter
[root@rdma-dev-25 ~]$ echo $?
0
[root@rdma-dev-25 ~]$


[root@rdma-dev-26 ~]$ ibv_devinfo 
hca_id:	bnxt_re0
	transport:			InfiniBand (0)
	fw_ver:				212.0.106.0
	node_guid:			020a:f7ff:feea:cd90
	sys_image_guid:			020a:f7ff:feea:cd90
	vendor_id:			0x14e4
	vendor_part_id:			5652
	hw_ver:				0x4540
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

[root@rdma-dev-26 ~]$ ibv_srq_pingpong -g 0 -d bnxt_re0 -i 1 172.31.40.125
  local address:  LID 0x0000, QPN 0x001022, PSN 0x414c12, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001023, PSN 0xfb6ccb, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001024, PSN 0xeac283, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001025, PSN 0x2aafdf, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001026, PSN 0xab653d, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001027, PSN 0x4ccb25, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001028, PSN 0xc50aea, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001029, PSN 0xbaaa65, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102a, PSN 0x05b16f, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102b, PSN 0x7a508e, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102c, PSN 0xe38950, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102d, PSN 0xb08ada, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102e, PSN 0x757588, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102f, PSN 0xd0b15e, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001030, PSN 0xd3f0fa, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001031, PSN 0xf6796c, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001022, PSN 0xa4ff12, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001023, PSN 0x40a3cb, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001024, PSN 0xcd2d83, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001025, PSN 0xb43edf, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001026, PSN 0xb7483d, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001027, PSN 0x4e7225, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001028, PSN 0x8325ea, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001029, PSN 0xee2965, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102a, PSN 0x0cc46f, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102b, PSN 0x8f678e, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102c, PSN 0x885450, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102d, PSN 0x3cf9da, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102e, PSN 0xfab888, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102f, PSN 0xc0385e, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001030, PSN 0x1a6bfa, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001031, PSN 0x7ad86c, GID fe80::20a:f7ff:feea:cea0
8192000 bytes in 0.00 seconds = 76382.28 Mbit/sec
1000 iters in 0.00 seconds = 0.86 usec/iter
[root@rdma-dev-26 ~]$ echo $?
0
[root@rdma-dev-26 ~]$ 

Test results for libibverbs-utils on rdma-dev-26:
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ibv_devices
      PASS |      0 | ibv_devinfo
      PASS |      0 | ibv_srq_pingpong

Comment 16 errata-xmlrpc 2019-08-06 12:46:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2100

Note You need to log in before you can comment on or make changes to this bug.