Bug 507522

Summary: "failed to start listening VM" when implementing domain migration on kvm
Product: Red Hat Enterprise Linux 5 Reporter: Edward Wang <edwang>
Component: libvirtAssignee: Daniel Berrangé <berrange>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.4CC: berrange, hbrock, llim, tyan, virt-maint, xen-maint, yoyzhang, yshao
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-10-29 16:27:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/libvirt/qemu/test1.log
none
Log from target machine
none
test1.xml
none
migrate log from local none

Description Edward Wang 2009-06-23 06:44:10 UTC
Description of problem:
I created one domain on kvm and start it. Then, try to migrate the domain to another host(two hosts are all kvm hypervisor, AMD cpus), find that the migration failed with following errors:
"error: operation failed: failed to start listening VM" 

Version-Release number of selected component (if applicable):
- libvirt: 0.6.3-11.el5

How reproducible:
100%

Steps to Reproduce:
1. virsh define rhel5u3.xml
2. virsh start rhel5u3
3. virsh migrate --live rhel5u3 qemu+ssh://x.x.x.x/system (x.x.x.x is remote ip)
  
Actual results:
In step 3, migration failed with following error message:
"error: operation failed: failed to start listening VM"

Expected results:
Migration should succeed on kvm.

Additional info:
1, firewall of local & remote is stopped
2, system information:
[root@dhcp-66-70-66 ~]# uname -a
Linux dhcp-66-70-66.nay.redhat.com 2.6.18-152.el5 #1 SMP Wed Jun 3 18:57:00 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Comment 1 Daniel Veillard 2009-06-23 08:32:59 UTC
Seems the error message comes from kvm actually
can you add /var/log/libvirt/qemu/rhel5u3.log from the target machine
as an attachment ?

Daniel

Comment 2 zhanghaiyan 2009-06-23 08:45:51 UTC
This can reproduce on 
[root@dhcp-66-70-82 ~]# uname -a
Linux dhcp-66-70-82.nay.redhat.com 2.6.18-155.el5 #1 SMP Fri Jun 19 17:06:31 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Attached test1.log is /var/log/libvirt/qemu/test1.log from traget machine

Comment 3 zhanghaiyan 2009-06-23 08:46:46 UTC
Created attachment 349063 [details]
/var/log/libvirt/qemu/test1.log

Comment 4 Daniel Berrangé 2009-06-23 09:04:57 UTC
Could you also provide the test1.log from the source machine, and the XML config 'virsh dumpxml test1'

Comment 5 zhanghaiyan 2009-06-23 09:10:44 UTC
Sorry, the below one test1.log I provided should be log from source machine

Comment 6 zhanghaiyan 2009-06-23 09:12:59 UTC
Upload target.log which is /var/log/libvirt/qemu/test1.log from traget machine

Comment 7 zhanghaiyan 2009-06-23 09:13:42 UTC
Created attachment 349070 [details]
Log from target machine

Comment 8 zhanghaiyan 2009-06-23 09:15:52 UTC
Upload test1.xml from '#virsh dumpxml test1'

Comment 9 zhanghaiyan 2009-06-23 09:17:41 UTC
Created attachment 349072 [details]
test1.xml

Comment 10 Daniel Berrangé 2009-06-23 09:20:08 UTC
Here's the important error message from the target:

qemu: could not open disk image /var/lib/libvirt/images/test1.img

You are trying to migrate a guest whose disk is not accessible on the target machine. You need to storage guest disks on a network'd filesystem or shared/networked block device

Comment 11 Edward Wang 2009-06-24 08:47:43 UTC
Hi,

Reopen this bug base on following descriptions:

My testing environment is different from the one of yoyzhang. 
During my migration testing, there is a shared pool ("pool1" below) between local and remote host and the domain ("rhel5u3" below) is located on the shared pool. But the migration (with command "virsh migrate --live rhel5u3 qemu+ssh://x.x.x.x/system", x.x.x.x is remote host ip) failed with the error message "error: operation failed: failed to start listening VM". Following is the config information of local & remote host and you can get the log "/var/log/libvirt/qemu/rhel5u3.log" of local and remote in attachment "rhel5u3local.log" & "rhel5u3remote.log" respectively.

=== local ===
/// pool information ///
[virsh pool-list]
Name                 State      Autostart 
-----------------------------------------
pool1                active     no        

[root@dhcp-66-70-66 edwang]# virsh pool-dumpxml pool1
<pool type='netfs'>
  <name>pool1</name>
  <uuid>3e3fce45-4f53-4fa7-bb32-11f34168b82b</uuid>
  <capacity>161061273600</capacity>
  <allocation>87838425088</allocation>
  <available>73222848512</available>
  <source>
    <host name='10.66.71.226'/>
    <dir path='/vol/libvirt2/manual'/>
    <format type='auto'/>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

/// domain information ///
virsh dumpxml rhel5u3
<domain type='kvm' id='4'>
  <name>rhel5u3</name>
  <uuid>8f7d7e4c-2c44-49f8-9c59-09b609b00ebd</uuid>
  <memory>1572864</memory>
  <currentMemory>1572864</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <source file='/var/lib/libvirt/images/RHEL-Server-5.3-64.qcow2'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <interface type='bridge'>
      <mac address='0c:07:06:3e:f7:21'/>
      <source bridge='eth0'/>
      <target dev='vnet0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

=== remote ===
/// pool information ///
virsh pool-dumpxml pool1
<pool type='netfs'>
  <name>pool1</name>
  <uuid>3e3fce45-4f53-4fa7-bb32-11f34168b82b</uuid>
  <capacity>161061273600</capacity>
  <allocation>87838359552</allocation>
  <available>73222914048</available>
  <source>
    <host name='10.66.71.226'/>
    <dir path='/vol/libvirt2/manual'/>
    <format type='auto'/>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

Comment 12 Edward Wang 2009-06-24 08:58:31 UTC
Created attachment 349210 [details]
migrate log from local

Comment 13 Edward Wang 2009-06-24 09:03:03 UTC
log (/var/log/libvirt/qemu/rhel5u3.log) on REMOTE host is blank and bugzilla prevent the uploading action...