RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 618562 - P2p migrate fail with error 'Unknown failure'.
Summary: P2p migrate fail with error 'Unknown failure'.
Keywords:
Status: CLOSED DUPLICATE of bug 615941
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.0
Hardware: x86_64
OS: Linux
low
high
Target Milestone: rc
: 6.1
Assignee: Eric Blake
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 622960 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-27 09:34 UTC by wangyimiao
Modified: 2011-06-21 01:53 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-06-21 01:53:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description wangyimiao 2010-07-27 09:34:06 UTC 
Description of problem:
Migrate fail with error 'Unknown failure'.

Version-Release number of selected component (if applicable):
-libvirt-0.8.1-18.el6.x86_64
-qemu-img-0.12.1.2-2.100.el6.x86_64
-qemu-kvm-0.12.1.2-2.100.el6.x86_64
-kernel-2.6.32-44.2.el6.x86_64

How reproducible:
5/5

Steps to Reproduce:

1.Run that commands on host'10.66.93.211' and host '10.66.93.98'.
# iptables -F
# setenforce 0

2.#  ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
24:36:97:75:02:35:88:8b:cc:8d:3e:e4:ce:c5:d7:c6 root.redhat.com
The key's randomart image is:
+--[ RSA 2048]----+
|       ..+= .    |
|      . .o +     |
|   o ++.+        |
|    *.o=         |
|   + .  So       |
|    + o . E      |
|   o o . .       |
|    o            |
|                 |
+-----------------+

3.# ssh-copy-id -i ~/.ssh/id_rsa.pub root.93.98 
root.93.98's password: 
Now try logging into the machine, with "ssh 'root.93.98'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

4.Login to target host'10.66.93.98'.
#ls /root/.ssh/
authorized_keys  known_hosts


5.Migrate 'VM1' from host '10.66.93.211' to '10.66.93.98'.

# virsh migrate --p2p vm1 qemu+ssh://10.66.93.98/system
error: Unknown failure

6.# tail -5 /var/log/messages
Jul 27 07:30:33 dhcp-93-211 libvirtd: 07:30:33.451: error : storagePoolRefresh:859 : internal error storage pool is not active
Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed
Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 : Unknown failure
Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.532: error : qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration unexpectedly failed
Jul 27 07:32:09 dhcp-93-211 libvirtd: 07:32:09.835: error : remoteIO:9888 : Unknown failure

7.# cat /var/log/libvirt/qemu/vm1.log 
..............
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name vm1 -uuid f1cc1278-23d8-ff1d-d486-c00484bf5ac3 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/mnt/wang/yimwang/RHEL-Server-6-64-virtio.qcow2,if=none,id=drive-ide0-0-0,boot=on,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c2:9f:c7,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 
char device redirected to /dev/pts/3


Actual results:
Migrate fail with error 'Unknown failure'.

Expected results:
Migrate successfully.
Comment 2 RHEL Program Management 2010-07-27 09:57:57 UTC 
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 3 Daniel Berrangé 2010-07-27 16:58:38 UTC 
> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.552: error :
> qemuDomainWaitForMigrationComplete:5371 : operation failed: Migration
> unexpectedly failed

This indicates that qemu's  'info migrate' command reported a failure. QEMU doesn't give us any further information on this, unless there is some in its stderr logfile.

> Jul 27 07:31:13 dhcp-93-211 libvirtd: 07:31:13.855: error : remoteIO:9888 :
> Unknown failure

This is likely just libvirt overwriting the original error message during cleanup after failure.
Comment 4 Eric Blake 2010-08-09 20:31:28 UTC 
I'm looking for a bit more information in an effort to reproduce this particular error message.  Did you run 'iptables -F' and 'setenforce 0' on the source (10.66.93.211), destination (10.66.93.98), or both machines?  Does the failure still happen with the latest nightly snapshots of each of the packages?  Do you need to run 'iptables -F', or is it sufficient to use system-config-firewall to allow traffic on ports 49152-49215)?  Did you get any SELinux AVC denial messages from setroubleshoot, or is disabling SELinux not necessary?

If it is just a quality of error message, then I'm inclined to defer any fix to 6.1.  If it is a real bug in migration, then we need to figure out if it is a bug in qemu or in libvirt.  At any rate, I still have to convince myself whether I've found the real cause of the error message, and I haven't been able to get to that point yet.
Comment 5 Eric Blake 2010-08-10 04:00:19 UTC 
I am seeing an issue where iptables is not accurately reflecting the firewall settings of system-config-firewall; but that may be due to bug 617505.  Ultimately, the firewall should not need to be relaxed if iptables is configured to allow tcp ports 49152-49215 through, but I'm having a hard time getting that configuration to stick in my setup.  In the meantime, the action of 'iptables -F' works around this issue.

I was able to successfully migrate between two rhel hosts, with 'iptables -F' and 'setenforcing 1', and these versions:
kernel-2.6.32-59.1.el6.x86_64
iptables-1.4.7-3.el6.x86_64
qemu-kvm-0.12.1.2-2.108.el6.x86_64
libvirt-0.8.1-21.el6.x86_64

Since 6.0 can perform a migration (provided all the configuration is correct), it is more of a documentation issue at this late stage in the game, rather than any further software improvements.  As for the quality of error messages, fixing that should be deferred to 6.1.
Comment 6 Eric Blake 2010-08-11 15:44:46 UTC 
*** Bug 622960 has been marked as a duplicate of this bug. ***
Comment 7 Eric Blake 2010-08-13 14:16:43 UTC 
Chris identified two bugs with upstream patches:
https://www.redhat.com/archives/libvir-list/2010-August/msg00215.html
https://www.redhat.com/archives/libvir-list/2010-August/msg00218.html
Comment 8 Eric Blake 2010-08-13 14:38:21 UTC 
Actually, Chris's patches fix --tunnelled, not --p2p; the two use slightly different code paths.  And the work-around for --tunnelled if Chris's patches are not backported is to run qemu-kvm as root:root instead of qemu:qemu, but that has its own set of drawbacks.
Comment 11 Suzanne Logcher 2011-03-28 21:11:13 UTC 
Since RHEL 6.1 External Beta has begun, and this bug remains 
unresolved, it has been rejected as it is not proposed as an 
exception or blocker.

Red Hat invites you to ask your support representative to 
propose this request, if appropriate and relevant, in the 
next release of Red Hat Enterprise Linux.
Comment 12 Dave Allan 2011-06-21 01:53:24 UTC 

*** This bug has been marked as a duplicate of bug 615941 ***
Note You need to log in before you can comment on or make changes to this bug.