RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1285044 - migration/RDMA: Race condition
Summary: migration/RDMA: Race condition
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Dr. David Alan Gilbert
QA Contact: xianwang
URL:
Whiteboard:
Depends On:
Blocks: 1475751
TreeView+ depends on / blocked
 
Reported: 2015-11-24 18:04 UTC by Dr. David Alan Gilbert
Modified: 2018-04-11 00:09 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-rhev-2.10.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1475751 (view as bug list)
Environment:
Last Closed: 2018-04-11 00:09:32 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1104 0 normal SHIPPED_LIVE Important: qemu-kvm-rhev security, bug fix, and enhancement update 2018-04-10 22:54:38 UTC

Description Dr. David Alan Gilbert 2015-11-24 18:04:27 UTC 
Description of problem:
There's a race condition in RDMA startup, triggerable by adding a ~2ms usleep before the post_recv_control(rdma, RDMA_WRID_READY) near the end of qemu_rdma_connect

Version-Release number of selected component (if applicable):
tested on 2.5-rc upstream

How reproducible:
100%ish

Steps to Reproduce:
1.
2.
3.

Actual results:

ibv_poll_cq wc.status=13 RNR retry counter exceeded!

This is on  the poll waiting for data in qemu_rdma_get_buffer
Expected results:
No error

Additional info:
I think this is because we're doing an rdma_post_send_control and sending a WRID_SEND_CONTROL dest->source before the source has post_recv_control - there needs to be some interlock there to ensure it happens in the right order.
Comment 1 Dr. David Alan Gilbert 2017-06-20 14:35:21 UTC 
possibly the race we're hitting on rxe?
Comment 2 Dr. David Alan Gilbert 2017-06-30 16:22:47 UTC 
Not the race we're hitting on rxe.

Posted fix for this one upstream:
   migration/rdma: Fix race on source
Comment 3 Dr. David Alan Gilbert 2017-07-19 14:38:06 UTC 
Merged upstream:

9cf2bab2edca1e651eef migration/rdma: Fix race on source
3a0f2ceaedcf70ff79b6 migration: Close file on failed migration load
0b3c15f09715acd78063 migration/rdma: fix qemu_rdma_block_for_wrid error paths
9c98cfbe72b21d9d84b9 migration/rdma: Allow cancelling while waiting for wrid
482a33c53cbc9d2b0c47 migration/rdma: Safely convert control types
32bce196344772df8d68 migration/rdma: Send error during cancelling

(Note if we backport this to a 2.9 world we don't want 'Close file on failed')
Comment 9 xianwang 2017-12-04 02:04:31 UTC 
Hi, Dave,
I am confused about how to verify this bug, is this a test only bug? it just needs to test common function of rdma migration on P8 and x86? or if not, could you give me some guide to verify this bug? thanks
Comment 10 Dr. David Alan Gilbert 2017-12-04 14:38:00 UTC 
(In reply to xianwang from comment #9)
> Hi, Dave,
> I am confused about how to verify this bug, is this a test only bug? it just
> needs to test common function of rdma migration on P8 and x86? or if not,
> could you give me some guide to verify this bug? thanks

It's a race condition that's very difficult to trigger in practice; so if you just check that RDMA migration is still working OK then that's fine as a test.
Comment 12 xianwang 2017-12-27 02:13:12 UTC 
So, I think this bug is fixed.
Comment 14 errata-xmlrpc 2018-04-11 00:09:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104
Note You need to log in before you can comment on or make changes to this bug.