Bug 618562 - P2p migrate fail with error 'Unknown failure'.
P2p migrate fail with error 'Unknown failure'.
Status: CLOSED DUPLICATE of bug 615941
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.0
x86_64 Linux
low Severity high
: rc
: 6.1
Assigned To: Eric Blake
Virtualization Bugs
: RHELNAK
: 622960 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-27 05:34 EDT by wangyimiao
Modified: 2011-06-20 21:53 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-06-20 21:53:24 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description wangyimiao 2010-07-27 05:34:06 EDT
Description of problem:
Migrate fail with error 'Unknown failure'.

Version-Release number of selected component (if applicable):
-libvirt-0.8.1-18.el6.x86_64
-qemu-img-0.12.1.2-2.100.el6.x86_64
-qemu-kvm-0.12.1.2-2.100.el6.x86_64
-kernel-2.6.32-44.2.el6.x86_64

How reproducible:
5/5

Steps to Reproduce:

1.Run that commands on host'10.66.93.211' and host '10.66.93.98'.
# iptables -F
# setenforce 0

2.#  ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
24:36:97:75:02:35:88:8b:cc:8d:3e:e4:ce:c5:d7:c6 root@dhcp-93-211.nay.redhat.com
The key's randomart image is:
+--[ RSA 2048]----+
|       ..+= .    |
|      . .o +     |
|   o ++.+        |
|    *.o=         |
|   + .  So       |
|    + o . E      |
|   o o . .       |
|    o            |
|                 |
+-----------------+

3.# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.66.93.98 
root@10.66.93.98's password: 
Now try logging into the machine, with "ssh 'root@10.66.93.98'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

4.Login to target host'10.66.93.98'.
#ls /root/.ssh/
authorized_keys  known_hosts


5.Migrate 'VM1' from host '10.66.93.211' to '10.66.93.98'.

# virsh migrate --p2p vm1 qemu+ssh://10.66.93.98/system
error: Unknown failure

6.# tail -5 /var/log/messages
Jul 27 07:30:33 dhcp-93-211 libvirtd: 07:30:33.451: error : storagePoolRefresh:859 : internal error storage pool is not active
Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed
Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 : Unknown failure
Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.532: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed
Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.835: error : remoteIO:9888 : Unknown failure

7.# cat /var/log/libvirt/qemu/vm1.log 
..............
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name vm1 -uuid f1cc1278-23d8-ff1d-d486-c00484bf5ac3 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/mnt/wang/yimwang/RHEL-Server-6-64-virtio.qcow2,if=none,id=drive-ide0-0-0,boot=on,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c2:9f:c7,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 
char device redirected to /dev/pts/3


Actual results:
Migrate fail with error 'Unknown failure'.

Expected results:
Migrate successfully.
Comment 2 RHEL Product and Program Management 2010-07-27 05:57:57 EDT
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 3 Daniel Berrange 2010-07-27 12:58:38 EDT
> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error :
> qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration
> unexpectedly failed

This indicates that qemu's  'info migrate' command reported a failure. QEMU doesn't give us any further information on this, unless there is some in its stderr logfile.

> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 :
> Unknown failure

This is likely just libvirt overwriting the original error message during cleanup after failure.
Comment 4 Eric Blake 2010-08-09 16:31:28 EDT
I'm looking for a bit more information in an effort to reproduce this particular error message.  Did you run 'iptables -F' and 'setenforce 0' on the source (10.66.93.211), destination (10.66.93.98), or both machines?  Does the failure still happen with the latest nightly snapshots of each of the packages?  Do you need to run 'iptables -F', or is it sufficient to use system-config-firewall to allow traffic on ports 49152-49215)?  Did you get any SELinux AVC denial messages from setroubleshoot, or is disabling SELinux not necessary?

If it is just a quality of error message, then I'm inclined to defer any fix to 6.1.  If it is a real bug in migration, then we need to figure out if it is a bug in qemu or in libvirt.  At any rate, I still have to convince myself whether I've found the real cause of the error message, and I haven't been able to get to that point yet.
Comment 5 Eric Blake 2010-08-10 00:00:19 EDT
I am seeing an issue where iptables is not accurately reflecting the firewall settings of system-config-firewall; but that may be due to bug 617505.  Ultimately, the firewall should not need to be relaxed if iptables is configured to allow tcp ports 49152-49215 through, but I'm having a hard time getting that configuration to stick in my setup.  In the meantime, the action of 'iptables -F' works around this issue.

I was able to successfully migrate between two rhel hosts, with 'iptables -F' and 'setenforcing 1', and these versions:
kernel-2.6.32-59.1.el6.x86_64
iptables-1.4.7-3.el6.x86_64
qemu-kvm-0.12.1.2-2.108.el6.x86_64
libvirt-0.8.1-21.el6.x86_64

Since 6.0 can perform a migration (provided all the configuration is correct), it is more of a documentation issue at this late stage in the game, rather than any further software improvements.  As for the quality of error messages, fixing that should be deferred to 6.1.
Comment 6 Eric Blake 2010-08-11 11:44:46 EDT
*** Bug 622960 has been marked as a duplicate of this bug. ***
Comment 8 Eric Blake 2010-08-13 10:38:21 EDT
Actually, Chris's patches fix --tunnelled, not --p2p; the two use slightly different code paths.  And the work-around for --tunnelled if Chris's patches are not backported is to run qemu-kvm as root:root instead of qemu:qemu, but that has its own set of drawbacks.
Comment 11 Suzanne Yeghiayan 2011-03-28 17:11:13 EDT
Since RHEL 6.1 External Beta has begun, and this bug remains 
unresolved, it has been rejected as it is not proposed as an 
exception or blocker.

Red Hat invites you to ask your support representative to 
propose this request, if appropriate and relevant, in the 
next release of Red Hat Enterprise Linux.
Comment 12 Dave Allan 2011-06-20 21:53:24 EDT

*** This bug has been marked as a duplicate of bug 615941 ***

Note You need to log in before you can comment on or make changes to this bug.