Bug 823467

Summary: virt-manager and virsh command hangs while the migration is happening and stop network on target host.
Product: Red Hat Enterprise Linux 6 Reporter: lei wang <leiwan>
Component: virt-managerAssignee: virt-mgr-maint
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.3CC: acathrow, dallan, dyasny, dyuan, gsun, jdenemar, mzhan, rwu, weizhan, whuang, yupzhang, zhpeng
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 822839 Environment:
Last Closed: 2012-07-24 18:43:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 822839    
Bug Blocks:    

Description lei wang 2012-05-21 10:24:33 UTC
+++ This bug was initially created as a clone of Bug #822839 +++

Description of problem:
virt-manager and virsh command hangs while the migration is happening and stop network on target host.

Version-Release number of selected component (if applicable):
# rpm -qa |grep libvirt
libvirt-debuginfo-0.9.10-14.el6.x86_64
libvirt-0.9.10-20.el6.x86_64
libvirt-devel-0.9.10-20.el6.x86_64
libvirt-client-0.9.10-20.el6.x86_64
libvirt-python-0.9.10-20.el6.x86_64
# rpm -q virt-manager
virt-manager-0.9.0-14.el6.x86_64
# rpm -q python-virtinst
python-virtinst-0.600.0-8.el6.noarch
# uname -r
2.6.32-272.el6.x86_64
#

How reproducible:
100%

procadion 
Configure migration environment with below command on both src and dst host.
#iptables -F
#setenforce 1
#setsebool virt_use_nfs  on
#mount 10.66.90.121:/vol/S3/libvirtauto /var/lib/libvirt/images/ -o vers=3

Configure ssh environment
# ssh-keygen
# ssh-copy-id root@target ip

Steps to Reproduce:
1. Start a domain;
   # virsh start <domain>

2. Start a live migration;
   # virsh migrate --live <domain> qemu+ssh://host2/system

3. Open virt-manager connect two hosts for monitor the process of migration.

4. While the migration is happening, stop network on target host.

  
Actual results:
1.The migration command don't prompt any message and don't stop automatic if migration does not success, virsh command hangs.
2.virt-manager hangs

Expected results:
1.The migration should fail, and could prompt some message about what happened
2.The virt-manager works well

Additional info:
If start network on target host, the below message shown, and virt-manager disconnect from all hosts.

[root@test ~]# virsh migrate --live mig-10 qemu+ssh://leiwan.redhat.com/system
2012-05-18 06:24:09.019+0000: 25903: info : libvirt version: 0.9.10, package: 20.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-05-15-21:58:29, x86-007.build.bos.redhat.com)
2012-05-18 06:24:09.019+0000: 25903: warning : virKeepAliveSend:128 : Failed to send keepalive response to client 0x7f23c0002f50
error: operation failed: migration job: unexpectedly failed
error: Reconnected to the hypervisor

If use command '#virsh domjobabort <domain>' to cancel migration job, the virsh command could NOT stop directly.

virt-manager Log:

# virt-manager --debug
2012-05-18 14:49:19,603 (cli:71): virt-manager startup
2012-05-18 14:49:19,603 (virt-manager:292): Launched as: /usr/share/virt-manager/virt-manager.py --debug
2012-05-18 14:49:19,603 (virt-manager:293): GTK version: (2, 18, 9)
2012-05-18 14:49:19,603 (virt-manager:294): virtManager import: <module 'virtManager' from '/usr/share/virt-manager/virtManager/__init__.pyc'>
2012-05-18 14:49:19,684 (engine:555): No inspection thread because libguestfs is too old, not available, or libvirt is not thread safe.
2012-05-18 14:49:19,685 (engine:346): About to connect to uris ['qemu+ssh://root.com/system', 'qemu:///system']
2012-05-18 14:49:19,689 (virt-manager:350): Connected to already running instance.

# cat virt-manager.log
[Fri, 18 May 2012 15:00:40 virt-manager 25231] ERROR (engine:453) Could not refresh connection qemu+ssh://root.com/system.
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 440, in _tick
    conn.tick()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1418, in tick
    self.hostinfo = self.vmm.getInfo()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3175, in getInfo
    if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: End of file while reading data: : Input/output error
[Fri, 18 May 2012 15:00:40 virt-manager 25231] DEBUG (engine:454) Closing connection since libvirtd appears to have stopped.
[Fri, 18 May 2012 15:00:40 virt-manager 25231] ERROR (engine:453) Could not refresh connection qemu:///system.
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 440, in _tick
    conn.tick()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1418, in tick
    self.hostinfo = self.vmm.getInfo()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3175, in getInfo
    if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: Cannot write data: Broken pipe
[Fri, 18 May 2012 15:00:40 virt-manager 25231] DEBUG (engine:454) Closing connection since libvirtd appears to have stopped.
[Fri, 18 May 2012 15:00:40 virt-manager 25231] ERROR (cli:85) Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/manager.py", line 975, in vm_selected
    self.change_run_text(vm.hasSavedImage())
  File "/usr/share/virt-manager/virtManager/domain.py", line 1089, in hasSavedImage
    return self._backend.hasManagedSaveImage(0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 738, in hasManagedSaveImage
    if ret == -1: raise libvirtError ('virDomainHasManagedSaveImage() failed', dom=self)
libvirtError: Cannot write data: Broken pipe
None
[Fri, 18 May 2012 15:00:40 virt-manager 25231] ERROR (cli:85) Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/manager.py", line 912, in conn_state_changed
    self.vm_selected()
  File "/usr/share/virt-manager/virtManager/manager.py", line 975, in vm_selected
    self.change_run_text(vm.hasSavedImage())
  File "/usr/share/virt-manager/virtManager/domain.py", line 1089, in hasSavedImage
    return self._backend.hasManagedSaveImage(0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 738, in hasManagedSaveImage
    if ret == -1: raise libvirtError ('virDomainHasManagedSaveImage() failed', dom=self)
libvirtError: Cannot write data: Broken pipe
None
#