Bug 1545248

Summary: libbnxt_re: ib_send_lat tests fails in event mode
Product: Red Hat Enterprise Linux 7 Reporter: Selvin Xavier (Broadcom) <sxavier>
Component: rdma-coreAssignee: Jarod Wilson <jarod>
Status: CLOSED ERRATA QA Contact: Mike Stowell <mstowell>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.5CC: brcm-roce-dev.pdl, jshortt, lmiksik, mstowell, rdma-dev-team
Target Milestone: rc   
Target Release: 7.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rdma-core-15-6.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 17:53:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1442258    

Description Selvin Xavier (Broadcom) 2018-02-14 13:24:45 UTC 
Description of problem:

ib_send_lat tests fails while testing with "-e" option. It is working fine with the default options of the command.

Version-Release number of selected component (if applicable):


How reproducible:

Always

Steps to Reproduce:
1. Load bnxt_en and bnxt_re driver 
2. Install rdma-core 
3. Run following command on server
   ib_send_lat -d bnxt_re0 -x 2 -e

4. Run following command on client

 ib_send_lat -d bnxt_re0 -x 2 -e <server ip>

Actual results:

Test fails with following error
run_iter_lat: Couldn't post send: scnt=3, err=Cannot allocate memory

Expected results:

Test should pass.

Additional info:
Comment 2 Selvin Xavier (Broadcom) 2018-02-14 13:26:25 UTC 
The issue seen with ib_read_lat also.
Comment 3 Selvin Xavier (Broadcom) 2018-02-14 14:04:41 UTC 
The issue is observed mostly because of the assumption by ib_*_lat tests. After posting a WQE to the SQ, even if the interrupt is generated on RQ, the application assumes that the interrupt is on SQ CQ and starts polling SQ CQ. 

bnxt_re hardware will always generate an extra interrupt if the sequence is as follows.

1. wait for interrupt
2. In the interrupt handling, ARM CQ first
3. Reap the CQEs from the CQ

In case of ib_read_lat, RQ receives the interrupt first. The interrupt handling arms RQ CQ first. This generates a pending interrupt on RQ CQ. Then the polling will be done for the first interrurpt (RQ CQ). The application posts to SQ. The app expect an interrupt on SQ CQ now. But the previous pending interrupt on RQ is scheduled and the waiting app polls the SQ CQ and sometimes doesn't find any CQE (CQE is not generated yet). 

The following patch is required to avoid the extra interrupt and test passes with the same. The details about the problem is also available in the patch description.

https://github.com/linux-rdma/rdma-core/commit/0b72e02a1fee3e383fad0aa2e7e5cf8c6a06c6bf#diff-a4ec7cf19bf17ce5aa732df8993db434
Comment 8 errata-xmlrpc 2018-04-10 17:53:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0954