Bug 1374613
| Summary: | Migration fails with "info migration reply was missing return status" when storage insufficient on target | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Dan Zheng <dzheng> |
| Component: | libvirt | Assignee: | Jiri Denemark <jdenemar> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.3 | CC: | dyuan, dzheng, fjin, jdenemar, knarra, lizhu, lmiksik, rbalakri, xuzhang, yafu, zpeng |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-2.0.0-9.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-03 18:54:34 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1258386 | ||
| Attachments: | |||
|
Description
Dan Zheng
2016-09-09 08:29:57 UTC
Created attachment 1199332 [details]
libvirtd log on source machine
Created attachment 1199351 [details]
vm log on target host
Created attachment 1199352 [details]
libvirtd log on target host
Created attachment 1199361 [details]
new qemu log on target
Created attachment 1199362 [details]
new libvirtd log on target
Created attachment 1199364 [details]
new libvirtd log on source
Broken by
commit 2e7cea24355328102c40dd127329ddf47d55a3e2
Refs: v1.2.17-87-g2e7cea2
Author: Jiri Denemark <jdenemar>
AuthorDate: Thu Jul 2 21:46:56 2015 +0200
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Jul 10 11:47:13 2015 +0200
qemu: Use error from Finish instead of "unexpectedly failed"
When QEMU exits on destination during migration, the source reports
either success (if the failure happened at the very end) or unhelpful
"unexpectedly failed" error message. However, the Finish API called on
the destination may report a real error so let's use it instead of the
generic one.
Signed-off-by: Jiri Denemark <jdenemar>
which was backported to 7.2 for bug 1090093.
If migration fails, this bug can cause the real error to be overwritten with useless (and wrong) "info migration reply was missing return status" message making the real error hard (or even impossible) to diagnose. Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2016-September/msg00322.html *** Bug 1330548 has been marked as a duplicate of this bug. *** Test packages of the scratch build:
libvirt-2.0.0-9.el7_rc.cdb55b1e.x86_64
qemu-kvm-rhev-2.6.0-23.el7.x86_64
Run on local host:
# virsh migrate --live --copy-storage-all --domain avocado-vt-vm1 --desturi qemu+ssh://10.66.4.167/system
error: operation failed: migration of disk vda failed
# virsh list --all
Id Name State
----------------------------------------------------
1 avocado-vt-vm1 running
local libvirtd.log:
2016-09-13 01:45:08.800+0000: 15275: info : qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_EVENT: mon=0x7f3a68002ea0 event={"timestamp": {"seconds": 1473731108, "microseconds": 800316}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio-disk0", "len": 10737418240, "offset": 4907335680, "speed": 9223372036853727232, "type": "mirror", "error": "No space left on device"}}
..
2016-09-13 01:45:08.801+0000: 15279: error : qemuMigrationDriveMirrorReady:1866 : operation failed: migration of disk vda failed
remote libvirtd.log:
2016-09-13 01:45:09.213+0000: 22963: error : virSecuritySELinuxSetFileconHelper:920 : unable to set security context 'system_u:object_r:usr_t:s0' on '/usr/share/avocado/data/avocado-vt/images/jeos-23-64.qcow2': No space left on device
See more details in attachment.
So the scratch build is ok.
Created attachment 1200313 [details]
libvirtd log on source machine for scratch build
Created attachment 1200314 [details]
libvirtd log on target machine for scratch build
Created attachment 1200327 [details]
vm log on target host fro scratch build
I can reproduce this. verify with build: libvirt-2.0.0-9.el7.x86_64 step: 1. Make insufficient storage on target machine 2. create pool both on source and target 3. Run # virsh migrate --live --copy-storage-all --domain avocado-vt-vm1 --desturi qemu+ssh://$target_ip/system --verbose error: operation failed: migration of disk vda failed get "No space left on device" in libvirtd.log. move to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html |