RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 694095 - Migration fails when migrate guest from RHEL6.1 host to RHEL6 host with the same libvirt version
Summary: Migration fails when migrate guest from RHEL6.1 host to RHEL6 host with the s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.1
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Alex Williamson
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-04-06 12:54 UTC by Min Zhan
Modified: 2013-01-09 23:45 UTC (History)
14 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.159.el6
Doc Type: Bug Fix
Doc Text:
[this is a fix for a bug introduced during 6.1 development, not a change between 6.0.0 and 6.1]
Clone Of:
Environment:
Last Closed: 2011-05-19 11:24:05 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0534 0 normal SHIPPED_LIVE Important: qemu-kvm security, bug fix, and enhancement update 2011-05-19 11:20:36 UTC

Description Min Zhan 2011-04-06 12:54:04 UTC 
Description of problem:
For RHEL6 host and RHEL6.1 host, keep the libvirt is the same. such as libvirt-0.8.7-15.el6.x86_64. Then migrate a guest from RHEL6 to RHEL6.1, works well. But migrate from RHEL6.1 to RHEL6 host, it fails.

Version-Release number of selected component (if applicable):
Host A: RHEL6 (ip: 6.148)
# uname -a
Linux dhcp-65-173.nay.redhat.com 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
qemu-kvm-0.12.1.2-2.113.el6.x86_64
libvirt-0.8.7-15.el6.x86_64

Host B: RHEL6.1 (ip: 4.216)
# uname -a
Linux dhcp-4-216.nay.redhat.com 2.6.32-128.el6.x86_64 #1 SMP Mon Mar 28 21:55:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
qemu-kvm-0.12.1.2-2.153.el6.x86_64
libvirt-0.8.7-15.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. On both hosts
# getenforce
Enforcing
# setsebool -P virt_use_nfs 1
# iptables -F

Add each other's ip/hostname to /etc/hosts file.

2. setup nfs server on Host A:
# cat /etc/exports 
/var/lib/libvirt/images *(rw,no_root_squash)

# service nfs start

3. mount nfs server on both hosts
Host A/B:
# mkdir /var/lib/libvirt/migrate
# mount -o vers=3 10.66.6.148:/var/lib/libvirt/images /var/lib/libvirt/migrate

4. Start a guest, kake sure its image is in the nfs storage
# virsh dumpxml rhel55
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/migrate/vr-rhel5u5'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
...

# virsh start rhel55

5. Migrate the guest from RHEL6 host to RHEL6.1 Host, it succeed.
# virsh migrate rhel55 qemu+ssh://10.66.4.216/system

6. Migrate the guest from RHEL6.1 host to RHEL6 host, it fails.
[root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system
The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established.
RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2.
Are you sure you want to continue connecting (yes/no)? yes
root.6.148's password: 
error: Unknown failure
  
Actual results:

An error display on RHEL6.1 host.
[root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system
The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established.
RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2.
Are you sure you want to continue connecting (yes/no)? yes
root.6.148's password: 
error: Unknown failure

After this, guest disappear in RHEL6.1 host, and in RHEL6 host this guest status is shutoff.

On RHEL6 host:
# tail -f /var/log/libvirt/qemu/rhel55.log
...
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel55.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=28,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:5f:48:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -incoming tcp:0.0.0.0:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/6
Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0
load of migration failed
2011-04-06 19:39:56.473: shutting down

On RHEL6.1 host:
# tail -f /var/log/libvirt/qemu/rhel55.log
...
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel55.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:5f:48:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/4
Using CPU model "cpu64-rhel6"
2011-04-06 19:50:06.014: shutting down
2011-04-06 19:53:50.034: starting up



Expected results:

Migration should be success from RHEL6.1 to RHEL6 

Additional info:

On RHEL6.1 Host:
# tail -f /var/log/messages
...
Apr  6 19:54:48 localhost libvirtd: 19:54:48.573: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/cpu/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/cpuacct/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/cpuset/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/memory/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/devices/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/freezer/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/blkio/libvirt/qemu/rhel55/ (16)
Apr  6 19:54:48 localhost avahi-daemon[7973]: Withdrawing address record for fe80::fc54:ff:fe5f:4812 on vnet0.
Apr  6 19:54:48 localhost kernel: virbr0: port 2(vnet0) entering disabled state
Apr  6 19:54:48 localhost kernel: libfcoe_device_notification: NETDEV_UNREGISTER vnet0
Apr  6 19:54:48 localhost kernel: device vnet0 left promiscuous mode
Apr  6 19:54:48 localhost kernel: virbr0: port 2(vnet0) entering disabled state
Apr  6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed
Apr  6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed
Apr  6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed
Apr  6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed
Apr  6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed
Apr  6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed
Apr  6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed
Apr  6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed

On RHEL6 Host:
# tail -f /var/log/messages
...
Apr  6 19:38:59 dhcp-65-173 avahi-daemon[1536]: Withdrawing address record for fe80::fc54:ff:fe5f:4812 on vnet1.
Apr  6 19:38:59 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state
Apr  6 19:38:59 dhcp-65-173 kernel: device vnet1 left promiscuous mode
Apr  6 19:38:59 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state
Apr  6 19:39:51 dhcp-65-173 kernel: device vnet1 entered promiscuous mode
Apr  6 19:39:51 dhcp-65-173 libvirtd: 19:39:51.289: 4115: warning : qemudStartVMDaemon:3282 : Executing /usr/libexec/qemu-kvm
Apr  6 19:39:51 dhcp-65-173 kernel: virbr0: topology change detected, propagating
Apr  6 19:39:51 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering forwarding state
Apr  6 19:39:51 dhcp-65-173 libvirtd: 19:39:51.298: 4115: warning : qemudStartVMDaemon:3292 : Executing done /usr/libexec/qemu-kvm
Apr  6 19:39:51 dhcp-65-173 qemu-kvm: Could not find keytab file: /etc/qemu/krb5.tab: No such file or directory
Apr  6 19:39:51 dhcp-65-173 NetworkManager[1522]: <warn> /sys/devices/virtual/net/vnet1: couldn't determine device driver; ignoring...
Apr  6 19:39:52 dhcp-65-173 avahi-daemon[1536]: Registering new address record for fe80::fc54:ff:fe5f:4812 on vnet1.*.
Apr  6 19:39:56 dhcp-65-173 avahi-daemon[1536]: Withdrawing address record for fe80::fc54:ff:fe5f:4812 on vnet1.
Apr  6 19:39:56 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state
Apr  6 19:39:56 dhcp-65-173 kernel: device vnet1 left promiscuous mode
Apr  6 19:39:56 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state
Comment 1 Osier Yang 2011-04-07 13:10:28 UTC 
[ RHEl6.0 ]

[root@dhcp-65- src]# LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5nodefconfig -nodefaults  -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -incoming tcp:0.0.0.0:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -monitor stdio
char device redirected to /dev/pts/1
QEMU 0.12.1 monitor - type 'help' for more information
(qemu) Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0


[ RHEL6.1 ]

[root@localhost ~]# LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cffconfig -nodefaults -rtc base=utc -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -monitor stdiochar device redirected to /dev/pts/6
Using CPU model "cpu64-rhel6"
QEMU 0.12.1 monitor - type 'help' for more information
(qemu) migrate -d tcp:10.66.6.148:49152
(qemu) info migrate 
Migration status: failed
(qemu) migrate -d tcp:10.66.6.148:49152
(qemu) info migrate 
Migration status: completed
(qemu) 


Mzhan used same libvirt but different qemu-kvm on 6.1 and 6.0, though IMHO qemu-kvm doesn't intend to support migration between 6.0 and 6.1, but reassign to avoid missing bug.
Comment 2 Daniel Berrangé 2011-04-11 14:36:52 UTC 
> though IMHO
> qemu-kvm doesn't intend to support migration between 6.0 and 6.1, but reassign
> to avoid missing bug.

This isn't correct. Migration is *expected* work in this case - provided the VM is using '-M rhel-6.0.0' to ensure compatibility of VM ABI.
Comment 3 Osier Yang 2011-04-12 05:20:54 UTC 
reassign to qemu-kvm
Comment 4 Dor Laor 2011-04-12 07:03:01 UTC 
It does not seems like a qemu-kvm bug. It's true that migration is expected to work with -M rhel-6.0.0 flag but why keep the same libvirt version? This is wrong.

More over, the reported error from above:
"
Actual results:

An error display on RHEL6.1 host.
[root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system
The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established.
RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2.
Are you sure you want to continue connecting (yes/no)? yes
root.6.148's password: 
error: Unknown failure
"

Seems to indicate that the authentication is wrong and migration did not start.
I'm closing this one. If you like to reopen it, please re-test with qemu directly.
Comment 5 Osier Yang 2011-04-12 10:17:56 UTC 
> 
> An error display on RHEL6.1 host.
> [root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system
> The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established.
> RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2.
> Are you sure you want to continue connecting (yes/no)? yes
> root.6.148's password: 
> error: Unknown failure
> "
> 
> Seems to indicate that the authentication is wrong and migration did not start.
> I'm closing this one. If you like to reopen it, please re-test with qemu
> directly.

The authentication is fine, otherwise libvirt won't start the migration and report sensiable error, actually migration was started and completed successfully on source host, the problem happens on target host when loading the migration. Though I did the testing with qemu directly, It's better QE do confirm it.
Comment 6 Min Zhan 2011-04-12 12:28:26 UTC 
According to Comment #1, It tests with qemu directly and problem still exists. So I reopen this bug.
Comment 7 Dor Laor 2011-04-12 14:25:51 UTC 
ok, this looks like the problem that failed the migration (please send better description next time):

"(qemu) Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0"
Comment 8 Juan Quintela 2011-04-12 15:54:01 UTC 
It looks like piix4_pm changed position in the pci bus.  I have to try to reproduce.
Comment 17 Mike Cao 2011-04-20 09:45:50 UTC 
Reproduced this issue on qemu-kvm-0.12.1.2-2.158.el6
another host is rhel6.0.z

1.start guest on RHEL6.1
/usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5 -nodefconfig -nodefaults -rtc base=utc -boot c -drive file=/opt/rhel5u6_64.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:5f:48:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc :2 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -monitor stdio
2.start listenning port on RHEL6.0
<commandline> -incoming tcp:0:5888
3.do live migration
(qemu)migrate -d tcp:xxx:5888

Actual Results:
on qemu-kvm-0.12.1.2-2.158.el6,
(qemu) Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0
on qemu-kvm-0.12.1.2-2.159.el6,
after migration ,guest still works.

also tried guest with -M rhel5.4.0 ,rhel5.4.4,rhel5.5.0 .all migration could be completed .

Based on above this issue has been fixed ald .
change status to verified.
Comment 19 Eduardo Habkost 2011-05-05 19:09:08 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
[this is a fix for a bug introduced during 6.1 development, not a change between 6.0.0 and 6.1]
Comment 20 errata-xmlrpc 2011-05-19 11:24:05 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html
Comment 21 errata-xmlrpc 2011-05-19 13:02:38 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html
Note You need to log in before you can comment on or make changes to this bug.