Bug 591076 - migration will cause libvirtd crash
migration will cause libvirtd crash
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.0
All Linux
low Severity high
: rc
: ---
Assigned To: Daniel Berrange
Virtualization Bugs
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-05-11 07:37 EDT by Johnny Liu
Modified: 2010-07-02 15:33 EDT (History)
6 users (show)

See Also:
Fixed In Version: libvirt-0.8.1-5.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-07-02 15:33:19 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Johnny Liu 2010-05-11 07:37:59 EDT
Description of problem:
migration will cause libvirtd crash

Version-Release number of selected component (if applicable):
libvirt-0.8.1-2.el6.x86_64
qemu-kvm-0.12.1.2-2.51.el6.x86_64
kernel-2.6.32-24.el6.x86_64
# rpm -qa|grep selinux
selinux-policy-targeted-3.7.19-12.el6.noarch
selinux-policy-3.7.19-12.el6.noarch
libselinux-2.0.90-3.el6.x86_64
libselinux-python-2.0.90-3.el6.x86_64
libselinux-utils-2.0.90-3.el6.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Install RHEL6.0-20100508.n.0 on two machines.

2. The selinux is in enforcing mode on the two machines
# getenforce 
Enforcing

3. Mount a nfs directory including vm image to local mount point on both source and target hosts.
# mount -o context="system_u:object_r:svirt_image_t:s0" 10.66.90.113:/vol/xenimage /mnt

4. Define domain on source host
# virsh dumpxml demo
<domain type='kvm'>
  <name>demo</name>
  <uuid>a0e61e7b-3b15-dcc0-e487-7c787bd960a1</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu'/>
      <source file='/mnt/jialiu/demo.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:5e:49:1a'/>
      <source network='default'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
  </devices>
</domain>

5. Start domain on source host
# virsh start demo
Domain demo started
# virsh list --all
 Id Name                 State
----------------------------------
  1 demo                 running


6. Live Migrate domain from source host to target host
# virsh migrate --live demo qemu+ssh://10.66.70.86/system
root@10.66.70.86's password: 
error: internal error canonical hostname pointed to localhost, but this is not allowed
(10.66.70.86 is target host IP)

7. According to BZ#580827, there is a workaround, so I change /etch/hosts as following on source host and target host
Target Host:
# cat /etc/hosts
#127.0.0.1	dhcp-66-70-86.nay.redhat.com	localhost.localdomain localhost
127.0.0.1	localhost localhost.localdomain
10.66.70.86	dhcp-66-70-86.nay.redhat.com
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Source Host:
# cat /etc/hosts
#127.0.0.1	dhcp-66-70-178.nay.redhat.com	localhost.localdomain localhost
127.0.0.1       localhost localhost.localdomain
10.66.70.86	dhcp-66-70-86.nay.redhat.com
:1         localhost localhost.localdomain localhost6 localhost6.localdomain6

8. Migrate again.
# virsh migrate --live demo qemu+ssh://10.66.70.86/system
root@10.66.70.86's password: 
error: operation failed: Migration unexpectedly failed
# virsh migrate --live demo qemu+ssh://10.66.70.86/system
error: unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: Connection refused
error: failed to connect to the hypervisor
# virsh list --all
error: unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: Connection refused
error: failed to connect to the hypervisor
# service libvirtd status
libvirtd dead but pid file exists

9. Try to restart libvirtd, but still failed.
# service libvirtd restart
Stopping libvirtd daemon:                                  [FAILED]
Starting libvirtd daemon:                                  [  OK  ]
# service libvirtd status
libvirtd dead but pid file exists

10. Check /var/log/message, get the following error:
...
May 11 06:23:52 dhcp-66-70-178 kernel: libvirtd[2695]: segfault at 0 ip 000000395d6800f2 sp 00007fc8b6cdf5a8 error 4 in libc-2.12.so[395d600000+175000]
...
  
Actual results:
Migration failed, and it cause libvirtd crash, even can not be restart.

Expected results:
Migration should be done successfully.

Additional info:
Comment 2 RHEL Product and Program Management 2010-05-11 09:11:55 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 4 Dave Allan 2010-05-18 23:43:50 EDT
libvirt-0.8.1-5.el6 has been built in RHEL-6-candidate with the fix

Dave
Comment 7 zhanghaiyan 2010-06-13 02:00:46 EDT
Verified this bug PASS with libvirt-0.8.1-8.el6.x86_64, no crash is seen anymore
- kernel-2.6.32-33.el6.x86_64
- qemu-img-0.12.1.2-2.73.el6.x86_64
- qemu-kvm-0.12.1.2-2.73.el6.x86_64
Comment 8 releng-rhel@redhat.com 2010-07-02 15:33:19 EDT
Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.