Bug 877312

Summary: libvirtd segfaults in qemuDomainObjSaveJob
Product: [Community] Virtualization Tools Reporter: Richard W.M. Jones <rjones>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: jdenemar, jtomko, rbalakri
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-02 08:12:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Richard W.M. Jones 2012-11-16 08:41:40 UTC 
Description of problem:

Running the libguestfs test:
make -C tests/parallel check-slow

Version-Release number of selected component (if applicable):

libvirt-0.10.2.1-2.fc18.x86_64

How reproducible:

Unknown, but rare.

Steps to Reproduce:
1. make -C tests/parallel check-slow
  
Actual results:

Crashes in all sorts of ways.

Expected results:

Shouldn't crash.

Additional info:

Core was generated by `/usr/sbin/libvirtd --timeout=30'.
Program terminated with signal 11, Segmentation fault.
#0  qemuDomainObjSaveJob (obj=obj@entry=0x7f593c2bae10, driver=0x7f5f3407d9a0, 
    driver=0x7f5f3407d9a0) at qemu/qemu_domain.c:665
665	    if (!virDomainObjIsActive(obj)) {
Missing separate debuginfos, use: debuginfo-install cryptopp-5.6.1-8.fc18.x86_64 libpciaccess-0.13.1-2.fc18.x86_64
(gdb) t a a bt

Thread 11 (Thread 0x7f5f409dc700 (LWP 2238)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9d20)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f409dc700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 10 (Thread 0x7f5f429e0700 (LWP 2234)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9f60)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f429e0700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 9 (Thread 0x7f5f431e1700 (LWP 2233)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9e40)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f431e1700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 8 (Thread 0x7f5f3f9da700 (LWP 2240)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9d20)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f3f9da700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 7 (Thread 0x7f5f441e3700 (LWP 2231)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9b70)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f441e3700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 6 (Thread 0x7f5f4aa10840 (LWP 2230)):
#0  0x0000003577846ecf in _IO_vfprintf_internal (s=s@entry=0x7fff9ece6c40, 
    format=<optimized out>, 
    format@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", 
    ap=ap@entry=0x7fff9ece6e68) at vfprintf.c:1352
#1  0x0000003577909f41 in __GI___vasprintf_chk (
    result_ptr=result_ptr@entry=0x7fff9ece6dc8, flags=flags@entry=1, 
    format=format@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", 
    args=0x7fff9ece6e68, args@entry=0x0) at vasprintf_chk.c:66
#2  0x0000003e02672a44 in vasprintf (__ap=__ap@entry=0x0, 
    __fmt=__fmt@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", 
    __ptr=0x7fff9ece6dc8, __ptr@entry=0x7fff9ece6ce8)
    at /usr/include/bits/stdio2.h:210
#3  virVasprintf (strp=strp@entry=0x7fff9ece6dc8, 
    fmt=fmt@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", 
    list=list@entry=0x7fff9ece6e68) at util/util.c:1984
#4  0x0000003e02665377 in virLogVMessage (
    category=0x3e0284dc3d "file.util/event_poll.c", priority=1, 
    funcname=0x3e0284e2d0 <__func__.9148> "virEventPollMakePollFDs", 
    linenr=378, flags=0, fmt=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", 
    vargs=vargs@entry=0x7fff9ece6e68) at util/logging.c:723
#5  0x0000003e026657ea in virLogMessage (
    category=category@entry=0x3e0284dc3d "file.util/event_poll.c", 
    priority=priority@entry=1, 
    funcname=funcname@entry=0x3e0284e2d0 <__func__.9148> "virEventPollMakePollFDs", linenr=linenr@entry=378, flags=flags@entry=0, 
    fmt=fmt@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d")
    at util/logging.c:670
#6  0x0000003e0265f45d in virEventPollMakePollFDs (nfds=<synthetic pointer>)
    at util/event_poll.c:374
#7  virEventPollRunOnce () at util/event_poll.c:605
#8  0x0000003e0265e3c7 in virEventRunDefaultImpl () at util/event.c:247
#9  0x0000003e0274b10d in virNetServerRun (srv=srv@entry=0xefb1e0)
    at rpc/virnetserver.c:748
#10 0x000000000040c2c3 in main (argc=<optimized out>, argv=<optimized out>)
    at libvirtd.c:1339

Thread 5 (Thread 0x7f5f401db700 (LWP 2239)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9e40)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f401db700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 4 (Thread 0x7f5f421df700 (LWP 2235)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9b70)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f421df700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 3 (Thread 0x7f5f411dd700 (LWP 2237)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328)
    at util/threads-pthread.c:117
#2  0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9f60)
    at util/threadpool.c:103
#3  0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x0000003578407d15 in start_thread (arg=0x7f5f411dd700)
    at pthread_create.c:308
#5  0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 2 (Thread 0x7f5f419de700 (LWP 2236)):
#0  0x00000035778bb32d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000035778ebd74 in usleep (useconds=useconds@entry=200000)
    at ../sysdeps/unix/sysv/linux/usleep.c:32
#2  0x00007f5f3c2a153b in qemuProcessKill (driver=driver@entry=0x7f5f3407d9a0, 
    vm=vm@entry=0x7f59a1f4e710, flags=flags@entry=0)
    at qemu/qemu_process.c:3933
#3  0x00007f5f3c2da89d in qemuDomainDestroyFlags (dom=<optimized out>, 
    flags=<optimized out>) at qemu/qemu_driver.c:1952
#4  0x0000003e026ee111 in virDomainDestroyFlags (
    domain=domain@entry=0x7f597ca423c0, flags=1) at libvirt.c:2264
#5  0x000000000041502d in remoteDispatchDomainDestroyFlags (
    args=0x7f597ca42400, rerr=0x7f5f419ddc70, client=<optimized out>, 
    server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1329
#6  remoteDispatchDomainDestroyFlagsHelper (server=<optimized out>, 
    client=<optimized out>, msg=<optimized out>, rerr=0x7f5f419ddc70, 
    args=0x7f597ca42400, ret=<optimized out>) at remote_dispatch.h:1307
#7  0x0000003e0274e632 in virNetServerProgramDispatchCall (msg=0xf241f0, 
    client=0xf22880, server=0xefb1e0, prog=0xf1cbb0)
    at rpc/virnetserverprogram.c:431
#8  virNetServerProgramDispatch (prog=0xf1cbb0, server=server@entry=0xefb1e0, 
    client=0xf22880, msg=0xf241f0) at rpc/virnetserverprogram.c:304
#9  0x0000003e0274a761 in virNetServerProcessMsg (msg=<optimized out>, 
    prog=<optimized out>, client=<optimized out>, srv=0xefb1e0)
    at rpc/virnetserver.c:170
#10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0xefb1e0)
    at rpc/virnetserver.c:191
#11 0x0000003e0266fafe in virThreadPoolWorker (opaque=opaque@entry=0xee9e40)
    at util/threadpool.c:144
#12 0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#13 0x0000003578407d15 in start_thread (arg=0x7f5f419de700)
    at pthread_create.c:308
#14 0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

Thread 1 (Thread 0x7f5f439e2700 (LWP 2232)):
#0  qemuDomainObjSaveJob (obj=obj@entry=0x7f593c2bae10, driver=0x7f5f3407d9a0, 
    driver=0x7f5f3407d9a0) at qemu/qemu_domain.c:665
#1  0x00007f5f3c28ec87 in qemuDomainObjBeginJobInternal (
    driver=driver@entry=0x7f5f3407d9a0, 
    driver_locked=driver_locked@entry=true, obj=obj@entry=0x7f593c2bae10, 
    job=job@entry=QEMU_JOB_DESTROY, 
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE) at qemu/qemu_domain.c:826
#2  0x00007f5f3c28fa6a in qemuDomainObjBeginJobWithDriver (
    driver=driver@entry=0x7f5f3407d9a0, obj=obj@entry=0x7f593c2bae10, 
    job=job@entry=QEMU_JOB_DESTROY) at qemu/qemu_domain.c:906
#3  0x00007f5f3c2da7d4 in qemuDomainDestroyFlags (dom=<optimized out>, 
    flags=<optimized out>) at qemu/qemu_driver.c:1970
#4  0x0000003e026ee111 in virDomainDestroyFlags (
    domain=domain@entry=0x7f59dfc6f220, flags=1) at libvirt.c:2264
#5  0x000000000041502d in remoteDispatchDomainDestroyFlags (
    args=0x7f59dfc6f260, rerr=0x7f5f439e1c70, client=<optimized out>, 
    server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1329
#6  remoteDispatchDomainDestroyFlagsHelper (server=<optimized out>, 
    client=<optimized out>, msg=<optimized out>, rerr=0x7f5f439e1c70, 
    args=0x7f59dfc6f260, ret=<optimized out>) at remote_dispatch.h:1307
#7  0x0000003e0274e632 in virNetServerProgramDispatchCall (msg=0xf1f6f0, 
    client=0xf23c40, server=0xefb1e0, prog=0xf1cbb0)
    at rpc/virnetserverprogram.c:431
#8  virNetServerProgramDispatch (prog=0xf1cbb0, server=server@entry=0xefb1e0, 
    client=0xf23c40, msg=0xf1f6f0) at rpc/virnetserverprogram.c:304
#9  0x0000003e0274a761 in virNetServerProcessMsg (msg=<optimized out>, 
    prog=<optimized out>, client=<optimized out>, srv=0xefb1e0)
    at rpc/virnetserver.c:170
#10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0xefb1e0)
    at rpc/virnetserver.c:191
#11 0x0000003e0266fafe in virThreadPoolWorker (opaque=opaque@entry=0xee9d20)
    at util/threadpool.c:144
#12 0x0000003e0266f589 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#13 0x0000003578407d15 in start_thread (arg=0x7f5f439e2700)
    at pthread_create.c:308
#14 0x00000035778f22cd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
Comment 1 Richard W.M. Jones 2012-11-16 08:44:35 UTC 
BTW:

(1) I had set NLDBG=4 because I was trying to reproduce another bug.

(2) I have libvirtd debugging enabled:

log_level=1
log_outputs="1:file:/tmp/libvirtd.log"
Comment 2 Jiri Denemark 2012-11-16 09:16:49 UTC 
Rich, since you had debugging enabled, could you attach the log? And that applies for the other bug as well.
Comment 3 Richard W.M. Jones 2012-11-16 09:33:51 UTC 
Unfortunately I've overwritten the log now.