Bug 972675 - Fail migration when VM get paused due to EIO
Fail migration when VM get paused due to EIO
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: rc
: ---
Assigned To: Peter Krempa
Virtualization Bugs
Depends On:
Blocks: 886416 961154 1045833
  Show dependency treegraph
Reported: 2013-06-10 07:51 EDT by Michal Skrivanek
Modified: 2013-12-22 07:25 EST (History)
9 users (show)

See Also:
Fixed In Version: libvirt-0.10.2-20.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1045833 (view as bug list)
Last Closed: 2013-11-21 04:02:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1581 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2013-11-20 20:11:35 EST

  None (edit)
Description Michal Skrivanek 2013-06-10 07:51:38 EDT
We should not support migration of VMs paused in EIO. According to QEMU a VM stuck in EIO is not safe to migrate and in risk of data corruption

It seems to be the most feasible way how to solve issues caused by losing storage connectivity in RHEV where we try to migrate all the VMs from the affected host, in reality due to race in status as seen by vdsm and engine and the actual VM state we do try to migrate VMs in EIO (they can get to EIO during migration as well) - this problem is only solvable at lower layer
If libvirt fails the migration at least at the end of it we can keep the paused VM on original host and resume once the storage is reconnected.
Comment 2 Daniel Berrange 2013-06-18 12:34:36 EDT
Sounds like it would be best for QEMU itself to refuse to accept the 'migrate' command if it is paused in EIO, and have it fail an ongoing migration if EIO occurs. Doing it in libvirt is somewhat racy since event notifications are asynchronous.
Comment 3 Peter Krempa 2013-07-03 09:00:34 EDT
The support for canceling ongoing migration was committed upstream a while ago in commits:

commit 5379bb0f33f1529f530a40958a10e8f02eb868bb
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Jun 12 16:11:22 2013 +0200

    migration: Don't propagate VIR_MIGRATE_ABORT_ON_ERROR
    This flag is meant for errors happening on the source of the migration
    and isn't used on the destination. To allow better migration
    compatibility, don't propagate it to the destination.

commit cf6d56ac433273b7e4e087bb861ebced0680cec3
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Jun 12 16:11:21 2013 +0200

    migration: Make erroring out on I/O error controllable by flag
    Paolo Bonzini pointed out that it's actually possible to migrate a qemu
    instance that was paused due to I/O error and it will be able to work on
    the destination if the storage is accessible.
    This patch introduces flag VIR_MIGRATE_ABORT_ON_ERROR that cancels the
    migration in case an I/O error happens while it's being performed and
    allows migration without this flag. This flag can be possibly used for
    other error reasons that may be introduced in the future.

commit 5f719f217ebf89668ca3c404e4b8288179c26c92
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Mon Jun 10 16:30:48 2013 +0200

    qemu: Forbid migration of machines with I/O errors
    Such machine can't be successuflly migrated unles the I/O error has
    recovered and might lead to data corruption. Forbid this kind of

commit caa467db626c8691d993e8e15d2cbb0bb043312c
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Mon Jun 10 16:05:45 2013 +0200

    qemu: Cancel migration if guest encoutners I/O error while migrating
    During a live migration the guest may receive a disk access I/O error.
    In this state the guest is unable to continue running on a remote host
    after migration as some state may be present in the kernel and not
    With this patch, the migration is canceled in such case so it can either
    continue on the source if the I/O issues are recovered or has to be
    destroyed anyways.
Comment 6 zhe peng 2013-07-23 01:44:13 EDT
verify with build :

 1:prepare two machine,source and target
 2:create a guest on source with shared nfs 
 test one:
 do migration after guest EIO:
 pause guest with I/O error
 #virsh domstate $guest --reason
 paused (I/O error)

 # virsh migrate --live spice qemu+ssh:// --verbose --unsafe
root@'s password: 
error: cannot open file '/var/lib/libvirt/migrate/xuzhang-Graph.img': Input/output error
 check guest state
 # virsh domstate $guest --reason
 paused (I/O error)
 check target, no guest exist

 test two:
 during migration guest receive EIO:
 # virsh domstate spice
 do migration
 # virsh migrate --live spice qemu+ssh:// --verbose --unsafe
root@'s password: 
 before migration finished, stop nfs server
Migration: [ 96 %]error: Unable to read from monitor: Connection reset by peer
the job stoped, check guest state
# virsh domstate spice --reason
paused (I/O error)
 check target, no guest exist. worked as expect, move to verified.
Comment 8 errata-xmlrpc 2013-11-21 04:02:39 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.