Red Hat Bugzilla – Bug 618562
P2p migrate fail with error 'Unknown failure'.
Last modified: 2011-06-20 21:53:24 EDT
Description of problem: Migrate fail with error 'Unknown failure'. Version-Release number of selected component (if applicable): -libvirt-0.8.1-18.el6.x86_64 -qemu-img-0.12.1.2-2.100.el6.x86_64 -qemu-kvm-0.12.1.2-2.100.el6.x86_64 -kernel-2.6.32-44.2.el6.x86_64 How reproducible: 5/5 Steps to Reproduce: 1.Run that commands on host'10.66.93.211' and host '10.66.93.98'. # iptables -F # setenforce 0 2.# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 24:36:97:75:02:35:88:8b:cc:8d:3e:e4:ce:c5:d7:c6 root@dhcp-93-211.nay.redhat.com The key's randomart image is: +--[ RSA 2048]----+ | ..+= . | | . .o + | | o ++.+ | | *.o= | | + . So | | + o . E | | o o . . | | o | | | +-----------------+ 3.# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.66.93.98 root@10.66.93.98's password: Now try logging into the machine, with "ssh 'root@10.66.93.98'", and check in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. 4.Login to target host'10.66.93.98'. #ls /root/.ssh/ authorized_keys known_hosts 5.Migrate 'VM1' from host '10.66.93.211' to '10.66.93.98'. # virsh migrate --p2p vm1 qemu+ssh://10.66.93.98/system error: Unknown failure 6.# tail -5 /var/log/messages Jul 27 07:30:33 dhcp-93-211 libvirtd: 07:30:33.451: error : storagePoolRefresh:859 : internal error storage pool is not active Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 : Unknown failure Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.532: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.835: error : remoteIO:9888 : Unknown failure 7.# cat /var/log/libvirt/qemu/vm1.log .............. LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name vm1 -uuid f1cc1278-23d8-ff1d-d486-c00484bf5ac3 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/mnt/wang/yimwang/RHEL-Server-6-64-virtio.qcow2,if=none,id=drive-ide0-0-0,boot=on,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c2:9f:c7,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 char device redirected to /dev/pts/3 Actual results: Migrate fail with error 'Unknown failure'. Expected results: Migrate successfully.
This issue has been proposed when we are only considering blocker issues in the current Red Hat Enterprise Linux release. ** If you would still like this issue considered for the current release, ask your support representative to file as a blocker on your behalf. Otherwise ask that it be considered for the next Red Hat Enterprise Linux release. **
> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error : > qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration > unexpectedly failed This indicates that qemu's 'info migrate' command reported a failure. QEMU doesn't give us any further information on this, unless there is some in its stderr logfile. > Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 : > Unknown failure This is likely just libvirt overwriting the original error message during cleanup after failure.
I'm looking for a bit more information in an effort to reproduce this particular error message. Did you run 'iptables -F' and 'setenforce 0' on the source (10.66.93.211), destination (10.66.93.98), or both machines? Does the failure still happen with the latest nightly snapshots of each of the packages? Do you need to run 'iptables -F', or is it sufficient to use system-config-firewall to allow traffic on ports 49152-49215)? Did you get any SELinux AVC denial messages from setroubleshoot, or is disabling SELinux not necessary? If it is just a quality of error message, then I'm inclined to defer any fix to 6.1. If it is a real bug in migration, then we need to figure out if it is a bug in qemu or in libvirt. At any rate, I still have to convince myself whether I've found the real cause of the error message, and I haven't been able to get to that point yet.
I am seeing an issue where iptables is not accurately reflecting the firewall settings of system-config-firewall; but that may be due to bug 617505. Ultimately, the firewall should not need to be relaxed if iptables is configured to allow tcp ports 49152-49215 through, but I'm having a hard time getting that configuration to stick in my setup. In the meantime, the action of 'iptables -F' works around this issue. I was able to successfully migrate between two rhel hosts, with 'iptables -F' and 'setenforcing 1', and these versions: kernel-2.6.32-59.1.el6.x86_64 iptables-1.4.7-3.el6.x86_64 qemu-kvm-0.12.1.2-2.108.el6.x86_64 libvirt-0.8.1-21.el6.x86_64 Since 6.0 can perform a migration (provided all the configuration is correct), it is more of a documentation issue at this late stage in the game, rather than any further software improvements. As for the quality of error messages, fixing that should be deferred to 6.1.
*** Bug 622960 has been marked as a duplicate of this bug. ***
Chris identified two bugs with upstream patches: https://www.redhat.com/archives/libvir-list/2010-August/msg00215.html https://www.redhat.com/archives/libvir-list/2010-August/msg00218.html
Actually, Chris's patches fix --tunnelled, not --p2p; the two use slightly different code paths. And the work-around for --tunnelled if Chris's patches are not backported is to run qemu-kvm as root:root instead of qemu:qemu, but that has its own set of drawbacks.
Since RHEL 6.1 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as an exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
*** This bug has been marked as a duplicate of bug 615941 ***