Bug 1378788 - Race condition during virtio-blk dataplane stop triggers "Virtqueue size exceeded"
Summary: Race condition during virtio-blk dataplane stop triggers "Virtqueue size exce...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Stefan Hajnoczi
QA Contact: lijin
URL:
Whiteboard:
Depends On:
Blocks: 1380320
TreeView+ depends on / blocked
 
Reported: 2016-09-23 09:29 UTC by Stefan Hajnoczi
Modified: 2020-01-17 15:57 UTC (History)
10 users (show)

Fixed In Version: QEMU 2.6
Doc Type: Bug Fix
Doc Text:
Due to a race condition in the virtio-blk dataplane, live migration of a guest in some cases failed with a "Virtqueue size exceeded" error message. This update prevents the race condition from occurring, and thus allows live migration to work more reliably.
Clone Of:
: 1380320 (view as bug list)
Environment:
Last Closed: 2016-11-07 21:36:23 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2673 0 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2016-11-08 01:06:13 UTC

Description Stefan Hajnoczi 2016-09-23 09:29:03 UTC 
Description of problem:
It is possible to trigger the "Virtqueue size exceeded" error on the source QEMU during live migration due to a race condition in virtio-blk dataplane.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-31.el7_2.22

How reproducible:
Non-deterministic.  Try 5-10 times.

Steps to Reproduce:

1. qemu-kvm -enable-kvm -m 1024 -cpu host -object iothread,id=thread0 -drive file=test.raw,if=none,id=drive0,format=raw,cache=none -device virtio-blk-pci,iothread=thread0,drive=drive0,id=virtio-blk0,bootindex=1

2. Log into the Linux guest.

3. Launch the destination QEMU: qemu-kvm -enable-kvm -m 1024 -cpu host -object iothread,id=thread0 -drive file=test.raw,if=none,id=drive0,format=raw,cache=none -device virtio-blk-pci,iothread=thread0,drive=drive0,id=virtio-blk0,bootindex=1 -incoming tcp::1234

4. Run "fio fio.job" inside the guest.  The contents of the fio.job file are:
[global]
filename=/dev/vda
ioengine=libaio
direct=1
runtime=60
ramp_time=5
gtod_reduce=1

[job]
readwrite=randread
iodepth=8
numjobs=8

3. While fio is running, migrate to the destination QEMU:
(qemu) migrate tcp:127.0.0.1:1234

Actual results:
Occassionally QEMU terminates with the "Virtqueue size exceeded" error message.

Expected results:
Live migration is successful and the fio benchmark continues running in the guest.

Additional info:
This bug does not affect RHEL 7.3 or recent upstream versions.
Comment 1 Stefan Hajnoczi 2016-09-23 09:31:05 UTC 
Note this bug only affects RHEL 7.2.z.  Please clone a z-stream bug.
Comment 4 huiqingding 2016-09-28 10:57:13 UTC 
I test this bug latest qemu-kvm-rhev:
qemu-kvm-rhev-2.6.0-28.el7.x86_64
3.10.0-510.el7.x86_64

The test steps are as comment #0. Did migration for 10+ times, the results are all pass. Live migration is successful and the fio benchmark continues running in the guest.
Comment 9 errata-xmlrpc 2016-11-07 21:36:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html
Note You need to log in before you can comment on or make changes to this bug.