Bug 694095
| Summary: | Migration fails when migrate guest from RHEL6.1 host to RHEL6 host with the same libvirt version | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Min Zhan <mzhan> |
| Component: | qemu-kvm | Assignee: | Alex Williamson <alex.williamson> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.1 | CC: | alex.williamson, bcao, berrange, dallan, dyuan, eblake, ehabkost, jyang, llim, mkenneth, mshao, mzhan, tburke, virt-maint |
| Target Milestone: | rc | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-0.12.1.2-2.159.el6 | Doc Type: | Bug Fix |
| Doc Text: |
[this is a fix for a bug introduced during 6.1 development, not a change between 6.0.0 and 6.1]
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-05-19 11:24:05 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[ RHEl6.0 ] [root@dhcp-65- src]# LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5nodefconfig -nodefaults -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -incoming tcp:0.0.0.0:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -monitor stdio char device redirected to /dev/pts/1 QEMU 0.12.1 monitor - type 'help' for more information (qemu) Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0 [ RHEL6.1 ] [root@localhost ~]# LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cffconfig -nodefaults -rtc base=utc -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -monitor stdiochar device redirected to /dev/pts/6 Using CPU model "cpu64-rhel6" QEMU 0.12.1 monitor - type 'help' for more information (qemu) migrate -d tcp:10.66.6.148:49152 (qemu) info migrate Migration status: failed (qemu) migrate -d tcp:10.66.6.148:49152 (qemu) info migrate Migration status: completed (qemu) Mzhan used same libvirt but different qemu-kvm on 6.1 and 6.0, though IMHO qemu-kvm doesn't intend to support migration between 6.0 and 6.1, but reassign to avoid missing bug. > though IMHO
> qemu-kvm doesn't intend to support migration between 6.0 and 6.1, but reassign
> to avoid missing bug.
This isn't correct. Migration is *expected* work in this case - provided the VM is using '-M rhel-6.0.0' to ensure compatibility of VM ABI.
reassign to qemu-kvm It does not seems like a qemu-kvm bug. It's true that migration is expected to work with -M rhel-6.0.0 flag but why keep the same libvirt version? This is wrong. More over, the reported error from above: " Actual results: An error display on RHEL6.1 host. [root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established. RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2. Are you sure you want to continue connecting (yes/no)? yes root.6.148's password: error: Unknown failure " Seems to indicate that the authentication is wrong and migration did not start. I'm closing this one. If you like to reopen it, please re-test with qemu directly.
>
> An error display on RHEL6.1 host.
> [root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system
> The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established.
> RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2.
> Are you sure you want to continue connecting (yes/no)? yes
> root.6.148's password:
> error: Unknown failure
> "
>
> Seems to indicate that the authentication is wrong and migration did not start.
> I'm closing this one. If you like to reopen it, please re-test with qemu
> directly.
The authentication is fine, otherwise libvirt won't start the migration and report sensiable error, actually migration was started and completed successfully on source host, the problem happens on target host when loading the migration. Though I did the testing with qemu directly, It's better QE do confirm it.
According to Comment #1, It tests with qemu directly and problem still exists. So I reopen this bug. ok, this looks like the problem that failed the migration (please send better description next time): "(qemu) Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0" It looks like piix4_pm changed position in the pci bus. I have to try to reproduce. Reproduced this issue on qemu-kvm-0.12.1.2-2.158.el6 another host is rhel6.0.z 1.start guest on RHEL6.1 /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5 -nodefconfig -nodefaults -rtc base=utc -boot c -drive file=/opt/rhel5u6_64.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:5f:48:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc :2 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -monitor stdio 2.start listenning port on RHEL6.0 <commandline> -incoming tcp:0:5888 3.do live migration (qemu)migrate -d tcp:xxx:5888 Actual Results: on qemu-kvm-0.12.1.2-2.158.el6, (qemu) Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0 on qemu-kvm-0.12.1.2-2.159.el6, after migration ,guest still works. also tried guest with -M rhel5.4.0 ,rhel5.4.4,rhel5.5.0 .all migration could be completed . Based on above this issue has been fixed ald . change status to verified.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
[this is a fix for a bug introduced during 6.1 development, not a change between 6.0.0 and 6.1]
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html |
Description of problem: For RHEL6 host and RHEL6.1 host, keep the libvirt is the same. such as libvirt-0.8.7-15.el6.x86_64. Then migrate a guest from RHEL6 to RHEL6.1, works well. But migrate from RHEL6.1 to RHEL6 host, it fails. Version-Release number of selected component (if applicable): Host A: RHEL6 (ip: 6.148) # uname -a Linux dhcp-65-173.nay.redhat.com 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux qemu-kvm-0.12.1.2-2.113.el6.x86_64 libvirt-0.8.7-15.el6.x86_64 Host B: RHEL6.1 (ip: 4.216) # uname -a Linux dhcp-4-216.nay.redhat.com 2.6.32-128.el6.x86_64 #1 SMP Mon Mar 28 21:55:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux qemu-kvm-0.12.1.2-2.153.el6.x86_64 libvirt-0.8.7-15.el6.x86_64 How reproducible: Always Steps to Reproduce: 1. On both hosts # getenforce Enforcing # setsebool -P virt_use_nfs 1 # iptables -F Add each other's ip/hostname to /etc/hosts file. 2. setup nfs server on Host A: # cat /etc/exports /var/lib/libvirt/images *(rw,no_root_squash) # service nfs start 3. mount nfs server on both hosts Host A/B: # mkdir /var/lib/libvirt/migrate # mount -o vers=3 10.66.6.148:/var/lib/libvirt/images /var/lib/libvirt/migrate 4. Start a guest, kake sure its image is in the nfs storage # virsh dumpxml rhel55 ... <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/migrate/vr-rhel5u5'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' unit='0'/> </disk> ... # virsh start rhel55 5. Migrate the guest from RHEL6 host to RHEL6.1 Host, it succeed. # virsh migrate rhel55 qemu+ssh://10.66.4.216/system 6. Migrate the guest from RHEL6.1 host to RHEL6 host, it fails. [root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established. RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2. Are you sure you want to continue connecting (yes/no)? yes root.6.148's password: error: Unknown failure Actual results: An error display on RHEL6.1 host. [root@dhcp-4-216 ~]# virsh migrate rhel55 qemu+ssh://10.66.6.148/system The authenticity of host '10.66.6.148 (10.66.6.148)' can't be established. RSA key fingerprint is d1:23:16:eb:96:ca:37:52:36:08:18:ca:e6:ad:28:b2. Are you sure you want to continue connecting (yes/no)? yes root.6.148's password: error: Unknown failure After this, guest disappear in RHEL6.1 host, and in RHEL6 host this guest status is shutoff. On RHEL6 host: # tail -f /var/log/libvirt/qemu/rhel55.log ... LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel55.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=28,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:5f:48:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -incoming tcp:0.0.0.0:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/6 Unknown savevm section or instance '0000:00:01.3/piix4_pm' 0 load of migration failed 2011-04-06 19:39:56.473: shutting down On RHEL6.1 host: # tail -f /var/log/libvirt/qemu/rhel55.log ... LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 7f21a3e9-924a-f4fa-c9ff-c27304e45cf5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel55.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/migrate/vr-rhel5u5,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:5f:48:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/4 Using CPU model "cpu64-rhel6" 2011-04-06 19:50:06.014: shutting down 2011-04-06 19:53:50.034: starting up Expected results: Migration should be success from RHEL6.1 to RHEL6 Additional info: On RHEL6.1 Host: # tail -f /var/log/messages ... Apr 6 19:54:48 localhost libvirtd: 19:54:48.573: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/cpu/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/cpuacct/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/cpuset/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/memory/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/devices/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/freezer/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost libvirtd: 19:54:48.576: 12573: error : virCgroupRemoveRecursively:668 : Unable to remove /cgroup/blkio/libvirt/qemu/rhel55/ (16) Apr 6 19:54:48 localhost avahi-daemon[7973]: Withdrawing address record for fe80::fc54:ff:fe5f:4812 on vnet0. Apr 6 19:54:48 localhost kernel: virbr0: port 2(vnet0) entering disabled state Apr 6 19:54:48 localhost kernel: libfcoe_device_notification: NETDEV_UNREGISTER vnet0 Apr 6 19:54:48 localhost kernel: device vnet0 left promiscuous mode Apr 6 19:54:48 localhost kernel: virbr0: port 2(vnet0) entering disabled state Apr 6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed Apr 6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed Apr 6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed Apr 6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed Apr 6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed Apr 6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed Apr 6 19:54:48 localhost lldpad[7907]: evb_ifdown:port vnet0 remove failed Apr 6 19:54:48 localhost lldpad[7907]: vdp_ifdown:vnet0 vdp data remove failed On RHEL6 Host: # tail -f /var/log/messages ... Apr 6 19:38:59 dhcp-65-173 avahi-daemon[1536]: Withdrawing address record for fe80::fc54:ff:fe5f:4812 on vnet1. Apr 6 19:38:59 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state Apr 6 19:38:59 dhcp-65-173 kernel: device vnet1 left promiscuous mode Apr 6 19:38:59 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state Apr 6 19:39:51 dhcp-65-173 kernel: device vnet1 entered promiscuous mode Apr 6 19:39:51 dhcp-65-173 libvirtd: 19:39:51.289: 4115: warning : qemudStartVMDaemon:3282 : Executing /usr/libexec/qemu-kvm Apr 6 19:39:51 dhcp-65-173 kernel: virbr0: topology change detected, propagating Apr 6 19:39:51 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering forwarding state Apr 6 19:39:51 dhcp-65-173 libvirtd: 19:39:51.298: 4115: warning : qemudStartVMDaemon:3292 : Executing done /usr/libexec/qemu-kvm Apr 6 19:39:51 dhcp-65-173 qemu-kvm: Could not find keytab file: /etc/qemu/krb5.tab: No such file or directory Apr 6 19:39:51 dhcp-65-173 NetworkManager[1522]: <warn> /sys/devices/virtual/net/vnet1: couldn't determine device driver; ignoring... Apr 6 19:39:52 dhcp-65-173 avahi-daemon[1536]: Registering new address record for fe80::fc54:ff:fe5f:4812 on vnet1.*. Apr 6 19:39:56 dhcp-65-173 avahi-daemon[1536]: Withdrawing address record for fe80::fc54:ff:fe5f:4812 on vnet1. Apr 6 19:39:56 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state Apr 6 19:39:56 dhcp-65-173 kernel: device vnet1 left promiscuous mode Apr 6 19:39:56 dhcp-65-173 kernel: virbr0: port 3(vnet1) entering disabled state