Bug 1116729

Summary:	Backport qemu_bh_schedule() race condition fix
Product:	Red Hat Enterprise Linux 7	Reporter:	Stefan Hajnoczi <stefanha>
Component:	qemu-kvm-rhev	Assignee:	John Snow <jsnow>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.1	CC:	huding, jsnow, juzhang, michen, mrezanin, rbalakri, sluo, stefanha, virt-bugs, virt-maint, xfu
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	qemu 2.1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1116728	Environment:
Last Closed:	2015-03-05 09:47:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1116728
Bug Blocks:

Description Stefan Hajnoczi 2014-07-07 07:53:43 UTC

+++ This bug was initially created as a clone of Bug #1116728 +++

There is a race condition in qemu_bh_schedule() that was fixed upstream:

commit 924fe1293c3e7a3c787bbdfb351e7f168caee3e9
Author: Stefan Hajnoczi <stefanha>
Date:   Tue Jun 3 11:21:01 2014 +0200

    aio: fix qemu_bh_schedule() bh->ctx race condition
    
    qemu_bh_schedule() is supposed to be thread-safe at least the first time
    it is called.  Unfortunately this is not quite true:
    
      bh->scheduled = 1;
      aio_notify(bh->ctx);
    
    Since another thread may run the BH callback once it has been scheduled,
    there is a race condition if the callback frees the BH before
    aio_notify(bh->ctx) has a chance to run.
    
    Reported-by: Stefan Priebe <s.priebe>
    Signed-off-by: Stefan Hajnoczi <stefanha>
    Reviewed-by: Paolo Bonzini <pbonzini>
    Tested-by: Stefan Priebe <s.priebe>

Upstream both block/rbd.c and block/gluster.c were affected by this race condition and users reported periodic crashes.  In rhev7 (based on QEMU 2.0.0) these block drivers are affected (unlike rhel7!).

Backporting this fix will improve the stability of guests using Gluster or RBD.  It may be hard to test this fix since the crash only happens rarely due to a race condition.

Comment 2 John Snow 2014-07-15 16:17:12 UTC

as noted, this was fixed upstream by 924fe1293c3e7a3c787bbdfb351e7f168caee3e9 so it will be included via the 2.1 rebase.

Comment 3 FuXiangChun 2014-07-22 10:03:34 UTC

Stefan,
would you please provide QE an effective way or reproducer to reproduce this bug? otherwise, QE don't know how to trigger/verify it.  Thanks!

Comment 4 Stefan Hajnoczi 2014-07-23 14:14:20 UTC

(In reply to FuXiangChun from comment #3)
> Stefan,
> would you please provide QE an effective way or reproducer to reproduce this
> bug? otherwise, QE don't know how to trigger/verify it.  Thanks!

There is no good test available for this race condition.  I think we'll have to rely on code review for this one.

Comment 5 FuXiangChun 2014-07-28 02:50:17 UTC

Stefan,
QE didn't find related patch from code. Can you confirm it?  This is my testing command. If it is wrong. please correct me. Thanks.

# rpm -qpi qemu-kvm-rhev-2.1.0-3.el7ev.preview.src.rpm --changelog|grep 1116729
nothing

Comment 6 Stefan Hajnoczi 2014-07-28 09:07:58 UTC

qemu-kvm-rhev-2.1.0-3.el7ev.preview.src.rpm is based on upstream an QEMU 2.1.0 release candidate.  It includes the relevant commit.  Since the commit comes directly from upstream instead of a backport, there is no Red Hat bugzilla information:

commit 924fe1293c3e7a3c787bbdfb351e7f168caee3e9
Author: Stefan Hajnoczi <stefanha>
Date:   Tue Jun 3 11:21:01 2014 +0200

    aio: fix qemu_bh_schedule() bh->ctx race condition

Looks good.

Comment 8 FuXiangChun 2014-08-28 13:41:25 UTC

verified with qemu-kvm-rhev-2.1.0-2.el7.src.rpm
double-check qemu_bh_schedule() in async.c source code.

void qemu_bh_schedule(QEMUBH *bh) 
{
    AioContext *ctx;

    if (bh->scheduled)
        return;
    ctx = bh->ctx;
    bh->idle = 0;
    /* Make sure that:
     * 1. idle & any writes needed by the callback are done before the
     *    locations are read in the aio_bh_poll.
     * 2. ctx is loaded before scheduled is set and the callback has a chance
     *    to execute.
     */
    smp_mb();
    bh->scheduled = 1;
    aio_notify(ctx);
}

According to comment6, This bug is fixed.

Comment 11 errata-xmlrpc 2015-03-05 09:47:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0624.html