Red Hat Bugzilla – Bug 618562
P2p migrate fail with error 'Unknown failure'.
Last modified: 2011-06-20 21:53:24 EDT
Description of problem:
Migrate fail with error 'Unknown failure'.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.Run that commands on host'10.66.93.211' and host '10.66.93.98'.
# iptables -F
# setenforce 0
2.# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
The key's randomart image is:
+--[ RSA 2048]----+
| ..+= . |
| . .o + |
| o ++.+ |
| *.o= |
| + . So |
| + o . E |
| o o . . |
| o |
3.# ssh-copy-id -i ~/.ssh/id_rsa.pub email@example.com
Now try logging into the machine, with "ssh 'firstname.lastname@example.org'", and check in:
to make sure we haven't added extra keys that you weren't expecting.
4.Login to target host'10.66.93.98'.
5.Migrate 'VM1' from host '10.66.93.211' to '10.66.93.98'.
# virsh migrate --p2p vm1 qemu+ssh://10.66.93.98/system
error: Unknown failure
6.# tail -5 /var/log/messages
Jul 27 07:30:33 dhcp-93-211 libvirtd: 07:30:33.451: error : storagePoolRefresh:859 : internal error storage pool is not active
Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed
Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 : Unknown failure
Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.532: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed
Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.835: error : remoteIO:9888 : Unknown failure
7.# cat /var/log/libvirt/qemu/vm1.log
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name vm1 -uuid f1cc1278-23d8-ff1d-d486-c00484bf5ac3 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/mnt/wang/yimwang/RHEL-Server-6-64-virtio.qcow2,if=none,id=drive-ide0-0-0,boot=on,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c2:9f:c7,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/3
Migrate fail with error 'Unknown failure'.
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.
** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error :
> qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration
> unexpectedly failed
This indicates that qemu's 'info migrate' command reported a failure. QEMU doesn't give us any further information on this, unless there is some in its stderr logfile.
> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 :
> Unknown failure
This is likely just libvirt overwriting the original error message during cleanup after failure.
I'm looking for a bit more information in an effort to reproduce this particular error message. Did you run 'iptables -F' and 'setenforce 0' on the source (10.66.93.211), destination (10.66.93.98), or both machines? Does the failure still happen with the latest nightly snapshots of each of the packages? Do you need to run 'iptables -F', or is it sufficient to use system-config-firewall to allow traffic on ports 49152-49215)? Did you get any SELinux AVC denial messages from setroubleshoot, or is disabling SELinux not necessary?
If it is just a quality of error message, then I'm inclined to defer any fix to 6.1. If it is a real bug in migration, then we need to figure out if it is a bug in qemu or in libvirt. At any rate, I still have to convince myself whether I've found the real cause of the error message, and I haven't been able to get to that point yet.
I am seeing an issue where iptables is not accurately reflecting the firewall settings of system-config-firewall; but that may be due to bug 617505. Ultimately, the firewall should not need to be relaxed if iptables is configured to allow tcp ports 49152-49215 through, but I'm having a hard time getting that configuration to stick in my setup. In the meantime, the action of 'iptables -F' works around this issue.
I was able to successfully migrate between two rhel hosts, with 'iptables -F' and 'setenforcing 1', and these versions:
Since 6.0 can perform a migration (provided all the configuration is correct), it is more of a documentation issue at this late stage in the game, rather than any further software improvements. As for the quality of error messages, fixing that should be deferred to 6.1.
*** Bug 622960 has been marked as a duplicate of this bug. ***
Chris identified two bugs with upstream patches:
Actually, Chris's patches fix --tunnelled, not --p2p; the two use slightly different code paths. And the work-around for --tunnelled if Chris's patches are not backported is to run qemu-kvm as root:root instead of qemu:qemu, but that has its own set of drawbacks.
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as an
exception or blocker.
Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
*** This bug has been marked as a duplicate of bug 615941 ***