Bug 506641

Summary: virsh migrate hangs for KVM guests
Product: Red Hat Enterprise Linux 5 Reporter: Nan Zhang <nzhang>
Component: libvirtAssignee: Daniel Berrangé <berrange>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: low    
Version: 5.4CC: gerrit.slomma, hbrock, llim, virt-maint, xen-maint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-10-29 16:25:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
migration hanging debug info
none
log of client migration
none
target host libvirtd with debug info none

Description Nan Zhang 2009-06-18 03:43:53 UTC
Created attachment 348372 [details]
migration hanging debug info

Description of problem:
Configure libvirtd with SASL authentication, it will be hung during migration connecting. See attached detailed debug info.

Version-Release number of selected component (if applicable):
libvirt 0.6.3-10.el5
rhel-5.4 (2.6.128-153.el5)

How reproducible:
Always

Steps to Reproduce:
Server:
# vi /etc/libvirt/libvirtd.conf
listen_tcp = 1
auth_tcp = "sasl"

# vi /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--listen"

# service libvirtd restart
Stopping libvirtd 

# saslpasswd2 -a libvirt root
Password:
Again (for verification):
# sasldblistusers2 -f /etc/libvirt/passwd.db
fred.redhat.com: userPassword
root.redhat.com: userPassword

Client:
# virsh migrate --live kvmtest qemu+tcp://10.66.70.85/system
Please enter your authentication name:root
Please enter your password:  
  
Actual results:
migration failed through SASL

Expected results:
migration successful with SASL authentication

Additional info:

Comment 1 Daniel Berrangé 2009-06-25 13:44:59 UTC
Please try the same operationg, using an SSH connection instead of SASL

# virsh migrate --live kvmtest qemu+ssh://root.70.85/system


If that also hangs, please provide logs of the client and target host libvirtd both with LIBVIRT_DEBUG=1 set.

Comment 2 Nan Zhang 2009-06-26 07:02:29 UTC
Yes, using an SSH connection also hangs, my current testing environment as following.

[root@dhcp-66-70-85 ~]# rpm -qa|grep 'kvm\|qemu\|kernel'|sort
etherboot-roms-kvm-5.4.4-10.el5
etherboot-zroms-kvm-5.4.4-10.el5
kernel-2.6.18-155.el5
kernel-xen-2.6.18-155.el5
kmod-kvm-83-80.el5
kvm-83-80.el5
kvm-qemu-img-83-80.el5

Comment 3 Nan Zhang 2009-06-26 07:07:35 UTC
Created attachment 349511 [details]
log of client migration

Comment 4 Nan Zhang 2009-06-26 07:08:29 UTC
Created attachment 349512 [details]
target host libvirtd with debug info

Comment 5 Daniel Berrangé 2009-06-26 11:17:12 UTC
At the time that it is hanging, can you see if any VM is running on the destination, eg

 virsh list --all


and grab the log file /var/log/libvirt/qemu/kvmtest.log from both the sopurce and destination host after it hangs.

It looks like the destation VM started up, reported success to libvirt, but then quickly shutdown again.

Comment 6 Nan Zhang 2009-06-29 13:07:34 UTC
Now, this issue cannot be reproduced every time, need to test further.

Comment 7 Gerrit Slomma 2009-07-24 21:32:46 UTC
I have a different error here:

[root@rr020 ~]# virsh migrate --live rr019v4 qemu+tcp://192.168.1.17/system
Please enter your authentication name:root
Please enter your password:
Fehler: Unknown failure

[root@rr017 ~]# tail -f /var/log/libvirt/qemu/rr019v4.log
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root /usr/libexec/qemu-kvm -S -M pc -m 1536 -smp 2 -name rr019v4 -uuid ff1ba599-5801-fca9-ab33-8962c2dfa46d -monitor pty -pidfile /var/run/libvirt/qemu//rr019v4.pid -boot c -drive file=/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-rr010:01-lun-7,if=ide,index=0,boot=on -net nic,macaddr=00:16:3e:69:94:06,vlan=0,model=e1000 -net tap,fd=17,script=,vlan=0,ifname=vnet0 -serial none -parallel none -usb -vnc 0.0.0.0:0 -k de -incoming tcp:0.0.0.0:49152
char device redirected to /dev/pts/1

[root@rr020 ~]# virsh migrate --live rr019v4 qemu+tcp://192.168.1.17/system tcp:192.168.1.17:4444
Please enter your authentication name:root
Please enter your password:
[root@rr020 ~]#

[root@rr017 ~]# tail -f /var/log/libvirt/qemu/rr019v4.log
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root /usr/libexec/qemu-kvm -S -M pc -m 1536 -smp 2 -name rr019v4 -uuid ff1ba599-5801-fca9-ab33-8962c2dfa46d -monitor pty -pidfile /var/run/libvirt/qemu//rr019v4.pid -boot c -drive file=/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-rr010:01-lun-7,if=ide,index=0,boot=on -net nic,macaddr=00:16:3e:69:94:06,vlan=0,model=e1000 -net tap,fd=17,script=,vlan=0,ifname=vnet0 -serial none -parallel none -usb -vnc 0.0.0.0:0 -k de -incoming tcp:0.0.0.0:4444
char device redirected to /dev/pts/1

I verified that this is no iptables-issue with disabling iptables.

on both hosts

Installed Packages
etherboot-zroms-kvm.x86_64   5.4.4-10.el5          installed
kernel.x86_64                2.6.18-155.el5        installed
kmod-kvm.x86_64              83-80.el5             installed
kvm.x86_64                   83-80.el5             installed
libvirt.x86_64               0.6.3-11.el5          installed

Comment 8 Hugh Brock 2009-10-29 16:25:49 UTC
Closing this since we were unable to reproduce it in testing (comment #7 is not relevant to this bug)