Bug 507522 - "failed to start listening VM" when implementing domain migration on kvm
"failed to start listening VM" when implementing domain migration on kvm
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt (Show other bugs)
5.4
All Linux
medium Severity medium
: rc
: ---
Assigned To: Daniel Berrange
Virtualization Bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-23 02:44 EDT by Edward Wang
Modified: 2009-12-14 16:09 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-10-29 12:27:15 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/libvirt/qemu/test1.log (4.73 KB, text/plain)
2009-06-23 04:46 EDT, zhanghaiyan
no flags Details
Log from target machine (1.63 KB, text/plain)
2009-06-23 05:13 EDT, zhanghaiyan
no flags Details
test1.xml (1.15 KB, text/xml)
2009-06-23 05:17 EDT, zhanghaiyan
no flags Details
migrate log from local (1.89 KB, text/plain)
2009-06-24 04:58 EDT, Edward Wang
no flags Details

  None (edit)
Description Edward Wang 2009-06-23 02:44:10 EDT
Description of problem:
I created one domain on kvm and start it. Then, try to migrate the domain to another host(two hosts are all kvm hypervisor, AMD cpus), find that the migration failed with following errors:
"error: operation failed: failed to start listening VM" 

Version-Release number of selected component (if applicable):
- libvirt: 0.6.3-11.el5

How reproducible:
100%

Steps to Reproduce:
1. virsh define rhel5u3.xml
2. virsh start rhel5u3
3. virsh migrate --live rhel5u3 qemu+ssh://x.x.x.x/system (x.x.x.x is remote ip)
  
Actual results:
In step 3, migration failed with following error message:
"error: operation failed: failed to start listening VM"

Expected results:
Migration should succeed on kvm.

Additional info:
1, firewall of local & remote is stopped
2, system information:
[root@dhcp-66-70-66 ~]# uname -a
Linux dhcp-66-70-66.nay.redhat.com 2.6.18-152.el5 #1 SMP Wed Jun 3 18:57:00 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Comment 1 Daniel Veillard 2009-06-23 04:32:59 EDT
Seems the error message comes from kvm actually
can you add /var/log/libvirt/qemu/rhel5u3.log from the target machine
as an attachment ?

Daniel
Comment 2 zhanghaiyan 2009-06-23 04:45:51 EDT
This can reproduce on 
[root@dhcp-66-70-82 ~]# uname -a
Linux dhcp-66-70-82.nay.redhat.com 2.6.18-155.el5 #1 SMP Fri Jun 19 17:06:31 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Attached test1.log is /var/log/libvirt/qemu/test1.log from traget machine
Comment 3 zhanghaiyan 2009-06-23 04:46:46 EDT
Created attachment 349063 [details]
/var/log/libvirt/qemu/test1.log
Comment 4 Daniel Berrange 2009-06-23 05:04:57 EDT
Could you also provide the test1.log from the source machine, and the XML config 'virsh dumpxml test1'
Comment 5 zhanghaiyan 2009-06-23 05:10:44 EDT
Sorry, the below one test1.log I provided should be log from source machine
Comment 6 zhanghaiyan 2009-06-23 05:12:59 EDT
Upload target.log which is /var/log/libvirt/qemu/test1.log from traget machine
Comment 7 zhanghaiyan 2009-06-23 05:13:42 EDT
Created attachment 349070 [details]
Log from target machine
Comment 8 zhanghaiyan 2009-06-23 05:15:52 EDT
Upload test1.xml from '#virsh dumpxml test1'
Comment 9 zhanghaiyan 2009-06-23 05:17:41 EDT
Created attachment 349072 [details]
test1.xml
Comment 10 Daniel Berrange 2009-06-23 05:20:08 EDT
Here's the important error message from the target:

qemu: could not open disk image /var/lib/libvirt/images/test1.img

You are trying to migrate a guest whose disk is not accessible on the target machine. You need to storage guest disks on a network'd filesystem or shared/networked block device
Comment 11 Edward Wang 2009-06-24 04:47:43 EDT
Hi,

Reopen this bug base on following descriptions:

My testing environment is different from the one of yoyzhang@redhat.com. 
During my migration testing, there is a shared pool ("pool1" below) between local and remote host and the domain ("rhel5u3" below) is located on the shared pool. But the migration (with command "virsh migrate --live rhel5u3 qemu+ssh://x.x.x.x/system", x.x.x.x is remote host ip) failed with the error message "error: operation failed: failed to start listening VM". Following is the config information of local & remote host and you can get the log "/var/log/libvirt/qemu/rhel5u3.log" of local and remote in attachment "rhel5u3local.log" & "rhel5u3remote.log" respectively.

=== local ===
/// pool information ///
[virsh pool-list]
Name                 State      Autostart 
-----------------------------------------
pool1                active     no        

[root@dhcp-66-70-66 edwang]# virsh pool-dumpxml pool1
<pool type='netfs'>
  <name>pool1</name>
  <uuid>3e3fce45-4f53-4fa7-bb32-11f34168b82b</uuid>
  <capacity>161061273600</capacity>
  <allocation>87838425088</allocation>
  <available>73222848512</available>
  <source>
    <host name='10.66.71.226'/>
    <dir path='/vol/libvirt2/manual'/>
    <format type='auto'/>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

/// domain information ///
virsh dumpxml rhel5u3
<domain type='kvm' id='4'>
  <name>rhel5u3</name>
  <uuid>8f7d7e4c-2c44-49f8-9c59-09b609b00ebd</uuid>
  <memory>1572864</memory>
  <currentMemory>1572864</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <source file='/var/lib/libvirt/images/RHEL-Server-5.3-64.qcow2'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <interface type='bridge'>
      <mac address='0c:07:06:3e:f7:21'/>
      <source bridge='eth0'/>
      <target dev='vnet0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

=== remote ===
/// pool information ///
virsh pool-dumpxml pool1
<pool type='netfs'>
  <name>pool1</name>
  <uuid>3e3fce45-4f53-4fa7-bb32-11f34168b82b</uuid>
  <capacity>161061273600</capacity>
  <allocation>87838359552</allocation>
  <available>73222914048</available>
  <source>
    <host name='10.66.71.226'/>
    <dir path='/vol/libvirt2/manual'/>
    <format type='auto'/>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>
Comment 12 Edward Wang 2009-06-24 04:58:31 EDT
Created attachment 349210 [details]
migrate log from local
Comment 13 Edward Wang 2009-06-24 05:03:03 EDT
log (/var/log/libvirt/qemu/rhel5u3.log) on REMOTE host is blank and bugzilla prevent the uploading action...

Note You need to log in before you can comment on or make changes to this bug.