Bug 1570393

Summary: [Broadcom CCX 7.6 FEAT]libbnxt_re: Add SRQ support on Broadcom adapters
Product: Red Hat Enterprise Linux 7 Reporter: Selvin Xavier (Broadcom) <sxavier>
Component: rdma-coreAssignee: Jarod Wilson <jarod>
Status: CLOSED ERRATA QA Contact: Afom T. Michael <tmichael>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: anantha.subramanyam, bhu, brcm-roce-dev.pdl, ddutile, mstowell, rdma-dev-team, selvin.xavier, sxavier
Target Milestone: rcKeywords: FutureFeature
Target Release: 7.7   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rdma-core-22-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 12:46:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1614004    

Description Selvin Xavier (Broadcom) 2018-04-22 14:42:12 UTC 
Description of problem:

SRQ support is not available in libbnxt_re. We are preparing the patches to be pushed upstream. This needs to be pulled in once it is accepted in upstream.


Steps to Reproduce:
1. Load bnxt_re drivers and install rdma-core with libbnxt_re
2. Run ibv_srq_ping_pong


Actual results:
 
The test fails now since the support is not available in the library.



Additional info:
Comment 3 Mike Stowell 2018-04-23 13:15:40 UTC 
Thanks for the test info in BZ description - adding qa_ack+
Comment 4 Don Dutile (Red Hat) 2018-04-23 18:20:57 UTC 
Selvin,
It cannot be pulled in until it is (minimally) in a for-next, or for-rc branch of rdma-core.
Please list the upstream commit-id's once it is pulled in.

Then Jarod can assess whether he can pull it into RHEL-7.6 rdma-core, or not.
Comment 5 Selvin Xavier (Broadcom) 2018-04-24 01:05:42 UTC 
(In reply to Don Dutile from comment #4)
> Selvin,
> It cannot be pulled in until it is (minimally) in a for-next, or for-rc
> branch of rdma-core.
> Please list the upstream commit-id's once it is pulled in.
> 
> Then Jarod can assess whether he can pull it into RHEL-7.6 rdma-core, or not.

Don,
 This is not submitted yet. My colleague would be pushing the patches to upstream this week. Once it is accepted, i will update this BZ with the commit id.

Thanks,
Selvin
Comment 6 Selvin Xavier (Broadcom) 2018-10-04 05:00:39 UTC 
Jarod,
 This commit is in rdma-core now. 

commit 602bb7ce1502c2e874eec6f1f677fd0539ceba01 - bnxt_re: Add shared receive queue support to broadcom's roce driver

Thanks,
Selvin
Comment 7 Don Dutile (Red Hat) 2018-10-04 22:46:49 UTC 
(In reply to Selvin Xavier from comment #6)
> Jarod,
>  This commit is in rdma-core now. 
> 
> commit 602bb7ce1502c2e874eec6f1f677fd0539ceba01 - bnxt_re: Add shared
> receive queue support to broadcom's roce driver
> 
> Thanks,
> Selvin

... are you sure it isn't in RHEL-7.6's rdma-core already?
Comment 8 Selvin Xavier (Broadcom) 2018-10-15 12:33:55 UTC 
(In reply to Don Dutile from comment #7)
> (In reply to Selvin Xavier from comment #6)
> > Jarod,
> >  This commit is in rdma-core now. 
> > 
> > commit 602bb7ce1502c2e874eec6f1f677fd0539ceba01 - bnxt_re: Add shared
> > receive queue support to broadcom's roce driver
> > 
> > Thanks,
> > Selvin
> 
> ... are you sure it isn't in RHEL-7.6's rdma-core already?


This was posted on last month Sept 17. See the posting message.

https://www.spinics.net/lists/linux-rdma/msg69055.html

Also, i have confirmed from the source rpms rdma-core-17.2-3.el7.src.rpm and rdma-core-19-3.el8.src.rpm that this is not yet available.
Comment 9 Jarod Wilson 2018-10-15 16:43:54 UTC 
I think this is likely 7.7 material with a z-stream backport now, but I'm not 100% certain on that. We'll also need a RHEL8 bug for it to get backported there as well.
Comment 10 Don Dutile (Red Hat) 2018-10-15 18:00:47 UTC 
(In reply to Jarod Wilson from comment #9)
> I think this is likely 7.7 material with a z-stream backport now, but I'm
> not 100% certain on that. We'll also need a RHEL8 bug for it to get
> backported there as well.

+1, *but*, a (7.6)z-stream backport would require QE to re-test bnxt-re, as that enables new functionality (srpt uses SRQ if available), thus, it's not a 'bug fix' for z-stream, but new functionality.
Comment 11 Jarod Wilson 2018-10-15 18:13:26 UTC 
Oh, right, this is new feature enablement, not a straight bug-fix, so probably no go on the z-stream backport for 7.6.
Comment 12 Selvin Xavier (Broadcom) 2018-10-16 12:07:45 UTC 
Created a BZ for 8.x
https://bugzilla.redhat.com/show_bug.cgi?id=1639692
Comment 14 Afom T. Michael 2019-05-29 22:21:58 UTC 
Marking verified as our test has passed on 3.10.0-1048.el7.x86_64. Also, the ibv_srq_pingpong test of our libibverbs-utils has passed.

[root@rdma-dev-25 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 Beta (Maipo)
[root@rdma-dev-25 ~]$ uname -r
3.10.0-1048.el7.x86_64
[root@rdma-dev-25 ~]$ ip a s bnxt_roce
6: bnxt_roce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 00:0a:f7:ea:ce:a0 brd ff:ff:ff:ff:ff:ff
    inet 172.31.40.125/24 brd 172.31.40.255 scope global noprefixroute dynamic bnxt_roce
       valid_lft 1679sec preferred_lft 1679sec
    inet6 fe80::20a:f7ff:feea:cea0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@rdma-dev-25 ~]$ ibv_devinfo
hca_id:	bnxt_re0
	transport:			InfiniBand (0)
	fw_ver:				212.0.106.0
	node_guid:			020a:f7ff:feea:cea0
	sys_image_guid:			020a:f7ff:feea:cea0
	vendor_id:			0x14e4
	vendor_part_id:			5652
	hw_ver:				0x4540
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

[root@rdma-dev-25 ~]$ rpm -qa | grep -E 'rdma|verbs'
libibverbs-22.1-1.el7.x86_64
librdmacm-utils-22.1-1.el7.x86_64
rdma-core-22.1-1.el7.x86_64
libibverbs-utils-22.1-1.el7.x86_64
kernel-kernel-infiniband-librdmacm-utils-0.1-53.noarch
kernel-kernel-infiniband-libibverbs-utils-0.1-36.noarch
librdmacm-22.1-1.el7.x86_64
kernel-kernel-infiniband-nfsordma-1.0-11.noarch
rdma-core-devel-22.1-1.el7.x86_64
[root@rdma-dev-25 ~]$ rpm -ql libibverbs | grep libbnxt_re
/usr/lib64/libibverbs/libbnxt_re-rdmav22.so
[root@rdma-dev-25 ~]$ ibv_srq_pingpong -g 0 -d bnxt_re0 -i 1
  local address:  LID 0x0000, QPN 0x001022, PSN 0xa4ff12, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001023, PSN 0x40a3cb, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001024, PSN 0xcd2d83, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001025, PSN 0xb43edf, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001026, PSN 0xb7483d, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001027, PSN 0x4e7225, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001028, PSN 0x8325ea, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001029, PSN 0xee2965, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102a, PSN 0x0cc46f, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102b, PSN 0x8f678e, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102c, PSN 0x885450, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102d, PSN 0x3cf9da, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102e, PSN 0xfab888, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x00102f, PSN 0xc0385e, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001030, PSN 0x1a6bfa, GID fe80::20a:f7ff:feea:cea0
  local address:  LID 0x0000, QPN 0x001031, PSN 0x7ad86c, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001022, PSN 0x414c12, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001023, PSN 0xfb6ccb, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001024, PSN 0xeac283, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001025, PSN 0x2aafdf, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001026, PSN 0xab653d, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001027, PSN 0x4ccb25, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001028, PSN 0xc50aea, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001029, PSN 0xbaaa65, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102a, PSN 0x05b16f, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102b, PSN 0x7a508e, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102c, PSN 0xe38950, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102d, PSN 0xb08ada, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102e, PSN 0x757588, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x00102f, PSN 0xd0b15e, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001030, PSN 0xd3f0fa, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001031, PSN 0xf6796c, GID fe80::20a:f7ff:feea:cd90
8192000 bytes in 0.00 seconds = 40604.71 Mbit/sec
1000 iters in 0.00 seconds = 1.61 usec/iter
[root@rdma-dev-25 ~]$ echo $?
0
[root@rdma-dev-25 ~]$


[root@rdma-dev-26 ~]$ ibv_devinfo 
hca_id:	bnxt_re0
	transport:			InfiniBand (0)
	fw_ver:				212.0.106.0
	node_guid:			020a:f7ff:feea:cd90
	sys_image_guid:			020a:f7ff:feea:cd90
	vendor_id:			0x14e4
	vendor_part_id:			5652
	hw_ver:				0x4540
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

[root@rdma-dev-26 ~]$ ibv_srq_pingpong -g 0 -d bnxt_re0 -i 1 172.31.40.125
  local address:  LID 0x0000, QPN 0x001022, PSN 0x414c12, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001023, PSN 0xfb6ccb, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001024, PSN 0xeac283, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001025, PSN 0x2aafdf, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001026, PSN 0xab653d, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001027, PSN 0x4ccb25, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001028, PSN 0xc50aea, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001029, PSN 0xbaaa65, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102a, PSN 0x05b16f, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102b, PSN 0x7a508e, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102c, PSN 0xe38950, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102d, PSN 0xb08ada, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102e, PSN 0x757588, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x00102f, PSN 0xd0b15e, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001030, PSN 0xd3f0fa, GID fe80::20a:f7ff:feea:cd90
  local address:  LID 0x0000, QPN 0x001031, PSN 0xf6796c, GID fe80::20a:f7ff:feea:cd90
  remote address: LID 0x0000, QPN 0x001022, PSN 0xa4ff12, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001023, PSN 0x40a3cb, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001024, PSN 0xcd2d83, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001025, PSN 0xb43edf, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001026, PSN 0xb7483d, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001027, PSN 0x4e7225, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001028, PSN 0x8325ea, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001029, PSN 0xee2965, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102a, PSN 0x0cc46f, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102b, PSN 0x8f678e, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102c, PSN 0x885450, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102d, PSN 0x3cf9da, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102e, PSN 0xfab888, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x00102f, PSN 0xc0385e, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001030, PSN 0x1a6bfa, GID fe80::20a:f7ff:feea:cea0
  remote address: LID 0x0000, QPN 0x001031, PSN 0x7ad86c, GID fe80::20a:f7ff:feea:cea0
8192000 bytes in 0.00 seconds = 76382.28 Mbit/sec
1000 iters in 0.00 seconds = 0.86 usec/iter
[root@rdma-dev-26 ~]$ echo $?
0
[root@rdma-dev-26 ~]$ 

Test results for libibverbs-utils on rdma-dev-26:
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ibv_devices
      PASS |      0 | ibv_devinfo
      PASS |      0 | ibv_srq_pingpong
Comment 16 errata-xmlrpc 2019-08-06 12:46:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2100