Red Hat Bugzilla – Bug 1374613
Migration fails with "info migration reply was missing return status" when storage insufficient on target
Last modified: 2016-11-03 14:54:34 EDT
Description of problem: When the target machine is storage insufficient, the migration fails with unreasonable message. Version-Release number of selected component (if applicable): libvirt-2.0.0-6.el7.x86_64 qemu-kvm-rhev-2.6.0-23.el7.x86_64 3.10.0-500.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Make insufficient storage on target machine. Below makes the avaible 5G in the target file system. # dd if=/dev/zero of=/usr/share/avocado/data/avocado-vt/images/occupied bs=1G count=26 26+0 records in 26+0 records out 27917287424 bytes (28 GB) copied, 348.599 s, 80.1 MB/s # df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/rhel-root 52403200 47705004 4698196 92% / devtmpfs 3885964 0 3885964 0% /dev tmpfs 3902124 88 3902036 1% /dev/shm tmpfs 3902124 204308 3697816 6% /run tmpfs 3902124 0 3902124 0% /sys/fs/cgroup /dev/loop0 1900368 6148 1772980 1% /srv/node/swiftloopback /dev/mapper/rhel-home 427172056 51067188 376104868 12% /home /dev/sda1 508588 222032 286556 44% /boot tmpfs 780428 16 780412 1% /run/user/42 tmpfs 780428 0 780428 0% /run/user/0 2. Create pool on target machine # virsh pool-create-as --name precreation_pool --type dir --target /usr/share/avocado/data/avocado-vt/images Pool precreation_pool created 3. Run command on source machine # virsh migrate --live --copy-storage-all --domain avocado-vt-vm1 --desturi qemu+ssh://10.66.4.167/system error: internal error: info migration reply was missing return status Actual results: Fail with unreasonable messages. See above. Expected results: Fail with reasonable messages, like "error: cannot allocate 31457280000 bytes in file '/usr/share/avocado/data/avocado-vt/images/jeos-23-64.qcow2': No space left on device" Additional info: See attachments for logs.
Created attachment 1199332 [details] libvirtd log on source machine
Created attachment 1199351 [details] vm log on target host
Created attachment 1199352 [details] libvirtd log on target host
Created attachment 1199361 [details] new qemu log on target
Created attachment 1199362 [details] new libvirtd log on target
Created attachment 1199364 [details] new libvirtd log on source
Broken by commit 2e7cea24355328102c40dd127329ddf47d55a3e2 Refs: v1.2.17-87-g2e7cea2 Author: Jiri Denemark <jdenemar@redhat.com> AuthorDate: Thu Jul 2 21:46:56 2015 +0200 Commit: Jiri Denemark <jdenemar@redhat.com> CommitDate: Fri Jul 10 11:47:13 2015 +0200 qemu: Use error from Finish instead of "unexpectedly failed" When QEMU exits on destination during migration, the source reports either success (if the failure happened at the very end) or unhelpful "unexpectedly failed" error message. However, the Finish API called on the destination may report a real error so let's use it instead of the generic one. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> which was backported to 7.2 for bug 1090093.
If migration fails, this bug can cause the real error to be overwritten with useless (and wrong) "info migration reply was missing return status" message making the real error hard (or even impossible) to diagnose.
Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2016-September/msg00322.html
*** Bug 1330548 has been marked as a duplicate of this bug. ***
Test packages of the scratch build: libvirt-2.0.0-9.el7_rc.cdb55b1e.x86_64 qemu-kvm-rhev-2.6.0-23.el7.x86_64 Run on local host: # virsh migrate --live --copy-storage-all --domain avocado-vt-vm1 --desturi qemu+ssh://10.66.4.167/system error: operation failed: migration of disk vda failed # virsh list --all Id Name State ---------------------------------------------------- 1 avocado-vt-vm1 running local libvirtd.log: 2016-09-13 01:45:08.800+0000: 15275: info : qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_EVENT: mon=0x7f3a68002ea0 event={"timestamp": {"seconds": 1473731108, "microseconds": 800316}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio-disk0", "len": 10737418240, "offset": 4907335680, "speed": 9223372036853727232, "type": "mirror", "error": "No space left on device"}} .. 2016-09-13 01:45:08.801+0000: 15279: error : qemuMigrationDriveMirrorReady:1866 : operation failed: migration of disk vda failed remote libvirtd.log: 2016-09-13 01:45:09.213+0000: 22963: error : virSecuritySELinuxSetFileconHelper:920 : unable to set security context 'system_u:object_r:usr_t:s0' on '/usr/share/avocado/data/avocado-vt/images/jeos-23-64.qcow2': No space left on device See more details in attachment. So the scratch build is ok.
Created attachment 1200313 [details] libvirtd log on source machine for scratch build
Created attachment 1200314 [details] libvirtd log on target machine for scratch build
Created attachment 1200327 [details] vm log on target host fro scratch build
I can reproduce this. verify with build: libvirt-2.0.0-9.el7.x86_64 step: 1. Make insufficient storage on target machine 2. create pool both on source and target 3. Run # virsh migrate --live --copy-storage-all --domain avocado-vt-vm1 --desturi qemu+ssh://$target_ip/system --verbose error: operation failed: migration of disk vda failed get "No space left on device" in libvirtd.log. move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html