Bug 688860

Summary: libvirtd crashed sometimes
Product: Red Hat Enterprise Linux 6 Reporter: Cui Chun <ccui>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.1CC: bugproxy, dyuan, eblake, jyang, laine, llim, yoyzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-21 05:48:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
libvirtd log none

Description Cui Chun 2011-03-18 09:29:32 UTC
Description of problem:

libvirtd is crashed when tried to start a guest sometimes.


Version-Release number of selected component (if applicable):

kernel-2.6.32-122.el6
libvirt-0.8.7-11.el6
emu-kvm-0.12.1.2-2.150.el6

How reproducible:

Only found it on one host and can not reproduce on other hosts.

(10.66.4.213)

Steps to Reproduce:
1.defined a guest from xml. (eth1 does not exist in fact)

<domain type='kvm'>
  <name>rstest_nic</name>
  <uuid>146d23be-85b8-421f-a347-94315c086c4c</uuid>
  <memory>262144</memory>
  <currentMemory>262144</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.1.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/default-kvm-dimage'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='direct'>
      <mac address='00:1a:4a:b3:fd:d1'/>
      <source dev='eth1' mode='vepa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>


 
2. start the guest

# virsh start rstest_nic
error: Failed to start domain rstest_nic
error: server closed connection:

3. check libvirtd status

# /etc/init.d/libvirtd status
libvirtd dead but pid file exists


4. replace eth1 with eth0 and restart the guest. libvirtd is ok. 

  
Actual results:

libvirtd dead

Expected results:

fix it

Additional info:  attached is libvirtd log

Comment 2 Cui Chun 2011-03-18 09:35:04 UTC
Created attachment 486190 [details]
libvirtd log

Comment 3 Laine Stump 2011-03-18 19:21:19 UTC
Can you try to attach gdb to libvirtd before triggering the crash, and get a traceback?

I'm suspicious that this crash may be more fallout of Bug 677724 / Bug 677725, but a traceback will help us to decide.

Also, on a freshly rebooted machine that has no virt-manager (or other monitoring application/script) running, try the same exercise - if it doesn't crash then this is almost surely the same problem.

(note that the problem isn't just due to eth1 not existing - I tried configuring a direct interface with a non-existent target device and an error message was logged (rather than a crash occurring))

Comment 4 Osier Yang 2011-03-21 03:31:41 UTC
I traced on ccui's box last Friday, the following is the call trace. The cause
is it pass a NULL "res_ifname" to "qemuAuditNetDevice" in "qemuPhysIfaceConnect", Though have not much knowledge about macvtap, don't known what's the best idea to fix it.

(gdb) bt
#0  0x00000038c7a80062 in __strlen_sse2 () from /lib64/libc.so.6
#1  0x00000038cde062fd in audit_encode_nv_string () from /lib64/libaudit.so.1
#2  0x0000000000462214 in qemuAuditNetDevice (vmDef=0x867830, netDef=0x86c060, 
    device=0x0, success=false) at qemu/qemu_audit.c:157
#3  0x0000000000454080 in qemuPhysIfaceConnect (def=0x867830, 
    conn=0x7fffe0000b50, driver=0x776880, net=0x86c060, 
    qemuCmdFlags=<value optimized out>, vmop=<value optimized out>)
    at qemu/qemu_command.c:131
#4  0x000000000045d957 in qemuBuildCommandLine (conn=0x7fffe0000b50, 
    driver=<value optimized out>, def=0x867830, monitor_chr=0x7fff00000000, 
    monitor_json=255, qemuCmdFlags=<value optimized out>, migrateFrom=0x0, 
    migrateFd=-1, current_snapshot=0x0, vmop=VIR_VM_OP_CREATE)
    at qemu/qemu_command.c:3447
#5  0x0000000000448c35 in qemudStartVMDaemon (conn=0x7fffe0000b50, 
    driver=0x776880, vm=0x85e3a0, migrateFrom=0x0, start_paused=false, 
    stdin_fd=-1, stdin_path=0x0, vmop=VIR_VM_OP_CREATE)
    at qemu/qemu_driver.c:3230
#6  0x000000000044c463 in qemudDomainObjStart (conn=0x7fffe0000b50, 
    driver=0x776880, vm=0x85e3a0, start_paused=false)
    at qemu/qemu_driver.c:6647
#7  0x000000000044ca15 in qemudDomainStartWithFlags (dom=0x7fffe40008e0, 
    flags=0) at qemu/qemu_driver.c:6693
#8  0x00000038d9e95066 in virDomainCreate (domain=0x7fffe40008e0)
---Type <return> to continue, or q <return> to quit---
    at libvirt.c:5130
#9  0x0000000000429e28 in remoteDispatchDomainCreate (
    server=<value optimized out>, client=<value optimized out>, 
    conn=0x7fffe0000b50, hdr=<value optimized out>, rerr=0x7fffea1fbb90, 
    args=<value optimized out>, ret=0x7fffea1fbc70) at remote.c:1225
#10 0x000000000042c8ca in remoteDispatchClientCall (server=0x6fe650, 
    client=0x7fffec001200, msg=0x7fffec0014b0) at dispatch.c:530
#11 remoteDispatchClientRequest (server=0x6fe650, client=0x7fffec001200, 
    msg=0x7fffec0014b0) at dispatch.c:408
#12 0x000000000041c5d8 in qemudWorker (data=0x7fffec000908) at libvirtd.c:1582
#13 0x00000038c86077e1 in start_thread () from /lib64/libpthread.so.0
#14 0x00000038c7ae5dcd in clone () from /lib64/libc.so.6

Comment 5 Cui Chun 2011-03-21 04:27:12 UTC
Please refer to comment 4

Comment 6 Laine Stump 2011-03-21 05:48:36 UTC
Ah, this has already been fixed in libvirt-0.8.7-12:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-March/msg00318.html

This was included with Bug 642785 as a followup patch (see Comment 14 of that bug).

*** This bug has been marked as a duplicate of bug 642785 ***

Comment 7 Cui Chun 2011-03-21 06:40:45 UTC
Yes, libvirtd keeps running when rechecked this issue with libvirt-0.8.7-13.el6.
It has been fixed

Comment 8 Osier Yang 2011-03-25 13:11:30 UTC
*** Bug 690805 has been marked as a duplicate of this bug. ***