RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1378788 - Race condition during virtio-blk dataplane stop triggers "Virtqueue size exceeded"
Summary: Race condition during virtio-blk dataplane stop triggers "Virtqueue size exce...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Stefan Hajnoczi
QA Contact: lijin
URL:
Whiteboard:
Depends On:
Blocks: 1380320
TreeView+ depends on / blocked
 
Reported: 2016-09-23 09:29 UTC by Stefan Hajnoczi
Modified: 2020-01-17 15:57 UTC (History)
10 users (show)

Fixed In Version: QEMU 2.6
Doc Type: Bug Fix
Doc Text:
Due to a race condition in the virtio-blk dataplane, live migration of a guest in some cases failed with a "Virtqueue size exceeded" error message. This update prevents the race condition from occurring, and thus allows live migration to work more reliably.
Clone Of:
: 1380320 (view as bug list)
Environment:
Last Closed: 2016-11-07 21:36:23 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2673 0 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2016-11-08 01:06:13 UTC

Description Stefan Hajnoczi 2016-09-23 09:29:03 UTC 
Description of problem:
It is possible to trigger the "Virtqueue size exceeded" error on the source QEMU during live migration due to a race condition in virtio-blk dataplane.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-31.el7_2.22

How reproducible:
Non-deterministic.  Try 5-10 times.

Steps to Reproduce:

1. qemu-kvm -enable-kvm -m 1024 -cpu host -object iothread,id=thread0 -drive file=test.raw,if=none,id=drive0,format=raw,cache=none -device virtio-blk-pci,iothread=thread0,drive=drive0,id=virtio-blk0,bootindex=1

2. Log into the Linux guest.

3. Launch the destination QEMU: qemu-kvm -enable-kvm -m 1024 -cpu host -object iothread,id=thread0 -drive file=test.raw,if=none,id=drive0,format=raw,cache=none -device virtio-blk-pci,iothread=thread0,drive=drive0,id=virtio-blk0,bootindex=1 -incoming tcp::1234

4. Run "fio fio.job" inside the guest.  The contents of the fio.job file are:
[global]
filename=/dev/vda
ioengine=libaio
direct=1
runtime=60
ramp_time=5
gtod_reduce=1

[job]
readwrite=randread
iodepth=8
numjobs=8

3. While fio is running, migrate to the destination QEMU:
(qemu) migrate tcp:127.0.0.1:1234

Actual results:
Occassionally QEMU terminates with the "Virtqueue size exceeded" error message.

Expected results:
Live migration is successful and the fio benchmark continues running in the guest.

Additional info:
This bug does not affect RHEL 7.3 or recent upstream versions.
Comment 1 Stefan Hajnoczi 2016-09-23 09:31:05 UTC 
Note this bug only affects RHEL 7.2.z.  Please clone a z-stream bug.
Comment 4 huiqingding 2016-09-28 10:57:13 UTC 
I test this bug latest qemu-kvm-rhev:
qemu-kvm-rhev-2.6.0-28.el7.x86_64
3.10.0-510.el7.x86_64

The test steps are as comment #0. Did migration for 10+ times, the results are all pass. Live migration is successful and the fio benchmark continues running in the guest.
Comment 9 errata-xmlrpc 2016-11-07 21:36:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html
Note You need to log in before you can comment on or make changes to this bug.