Description of problem: Running the libguestfs test: make -C tests/parallel check-slow Version-Release number of selected component (if applicable): libvirt-0.10.2.1-2.fc18.x86_64 How reproducible: Unknown, but rare. Steps to Reproduce: 1. make -C tests/parallel check-slow Actual results: Crashes in all sorts of ways. Expected results: Shouldn't crash. Additional info: Core was generated by `/usr/sbin/libvirtd --timeout=30'. Program terminated with signal 11, Segmentation fault. #0 qemuDomainObjSaveJob (obj=obj@entry=0x7f593c2bae10, driver=0x7f5f3407d9a0, driver=0x7f5f3407d9a0) at qemu/qemu_domain.c:665 665 if (!virDomainObjIsActive(obj)) { Missing separate debuginfos, use: debuginfo-install cryptopp-5.6.1-8.fc18.x86_64 libpciaccess-0.13.1-2.fc18.x86_64 (gdb) t a a bt Thread 11 (Thread 0x7f5f409dc700 (LWP 2238)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9d20) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f409dc700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 10 (Thread 0x7f5f429e0700 (LWP 2234)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9f60) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f429e0700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 9 (Thread 0x7f5f431e1700 (LWP 2233)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9e40) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f431e1700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 8 (Thread 0x7f5f3f9da700 (LWP 2240)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9d20) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f3f9da700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 7 (Thread 0x7f5f441e3700 (LWP 2231)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9b70) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f441e3700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 6 (Thread 0x7f5f4aa10840 (LWP 2230)): #0 0x0000003577846ecf in _IO_vfprintf_internal (s=s@entry=0x7fff9ece6c40, format=<optimized out>, format@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", ap=ap@entry=0x7fff9ece6e68) at vfprintf.c:1352 #1 0x0000003577909f41 in __GI___vasprintf_chk ( result_ptr=result_ptr@entry=0x7fff9ece6dc8, flags=flags@entry=1, format=format@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", args=0x7fff9ece6e68, args@entry=0x0) at vasprintf_chk.c:66 #2 0x0000003e02672a44 in vasprintf (__ap=__ap@entry=0x0, __fmt=__fmt@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", __ptr=0x7fff9ece6dc8, __ptr@entry=0x7fff9ece6ce8) at /usr/include/bits/stdio2.h:210 #3 virVasprintf (strp=strp@entry=0x7fff9ece6dc8, fmt=fmt@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", list=list@entry=0x7fff9ece6e68) at util/util.c:1984 #4 0x0000003e02665377 in virLogVMessage ( category=0x3e0284dc3d "file.util/event_poll.c", priority=1, funcname=0x3e0284e2d0 <__func__.9148> "virEventPollMakePollFDs", linenr=378, flags=0, fmt=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d", vargs=vargs@entry=0x7fff9ece6e68) at util/logging.c:723 #5 0x0000003e026657ea in virLogMessage ( category=category@entry=0x3e0284dc3d "file.util/event_poll.c", priority=priority@entry=1, funcname=funcname@entry=0x3e0284e2d0 <__func__.9148> "virEventPollMakePollFDs", linenr=linenr@entry=378, flags=flags@entry=0, fmt=fmt@entry=0x3e0284e080 "Prepare n=%d w=%d, f=%d e=%d d=%d") at util/logging.c:670 #6 0x0000003e0265f45d in virEventPollMakePollFDs (nfds=<synthetic pointer>) at util/event_poll.c:374 #7 virEventPollRunOnce () at util/event_poll.c:605 #8 0x0000003e0265e3c7 in virEventRunDefaultImpl () at util/event.c:247 #9 0x0000003e0274b10d in virNetServerRun (srv=srv@entry=0xefb1e0) at rpc/virnetserver.c:748 #10 0x000000000040c2c3 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1339 Thread 5 (Thread 0x7f5f401db700 (LWP 2239)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9e40) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f401db700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 4 (Thread 0x7f5f421df700 (LWP 2235)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb350, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fb9b in virThreadPoolWorker (opaque=opaque@entry=0xee9b70) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f421df700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 3 (Thread 0x7f5f411dd700 (LWP 2237)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x0000003e0266f756 in virCondWait (c=c@entry=0xefb3e8, m=m@entry=0xefb328) at util/threads-pthread.c:117 #2 0x0000003e0266fbbb in virThreadPoolWorker (opaque=opaque@entry=0xee9f60) at util/threadpool.c:103 #3 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x0000003578407d15 in start_thread (arg=0x7f5f411dd700) at pthread_create.c:308 #5 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 2 (Thread 0x7f5f419de700 (LWP 2236)): #0 0x00000035778bb32d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000035778ebd74 in usleep (useconds=useconds@entry=200000) at ../sysdeps/unix/sysv/linux/usleep.c:32 #2 0x00007f5f3c2a153b in qemuProcessKill (driver=driver@entry=0x7f5f3407d9a0, vm=vm@entry=0x7f59a1f4e710, flags=flags@entry=0) at qemu/qemu_process.c:3933 #3 0x00007f5f3c2da89d in qemuDomainDestroyFlags (dom=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:1952 #4 0x0000003e026ee111 in virDomainDestroyFlags ( domain=domain@entry=0x7f597ca423c0, flags=1) at libvirt.c:2264 #5 0x000000000041502d in remoteDispatchDomainDestroyFlags ( args=0x7f597ca42400, rerr=0x7f5f419ddc70, client=<optimized out>, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1329 #6 remoteDispatchDomainDestroyFlagsHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f5f419ddc70, args=0x7f597ca42400, ret=<optimized out>) at remote_dispatch.h:1307 #7 0x0000003e0274e632 in virNetServerProgramDispatchCall (msg=0xf241f0, client=0xf22880, server=0xefb1e0, prog=0xf1cbb0) at rpc/virnetserverprogram.c:431 #8 virNetServerProgramDispatch (prog=0xf1cbb0, server=server@entry=0xefb1e0, client=0xf22880, msg=0xf241f0) at rpc/virnetserverprogram.c:304 #9 0x0000003e0274a761 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0xefb1e0) at rpc/virnetserver.c:170 #10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0xefb1e0) at rpc/virnetserver.c:191 #11 0x0000003e0266fafe in virThreadPoolWorker (opaque=opaque@entry=0xee9e40) at util/threadpool.c:144 #12 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #13 0x0000003578407d15 in start_thread (arg=0x7f5f419de700) at pthread_create.c:308 #14 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Thread 1 (Thread 0x7f5f439e2700 (LWP 2232)): #0 qemuDomainObjSaveJob (obj=obj@entry=0x7f593c2bae10, driver=0x7f5f3407d9a0, driver=0x7f5f3407d9a0) at qemu/qemu_domain.c:665 #1 0x00007f5f3c28ec87 in qemuDomainObjBeginJobInternal ( driver=driver@entry=0x7f5f3407d9a0, driver_locked=driver_locked@entry=true, obj=obj@entry=0x7f593c2bae10, job=job@entry=QEMU_JOB_DESTROY, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE) at qemu/qemu_domain.c:826 #2 0x00007f5f3c28fa6a in qemuDomainObjBeginJobWithDriver ( driver=driver@entry=0x7f5f3407d9a0, obj=obj@entry=0x7f593c2bae10, job=job@entry=QEMU_JOB_DESTROY) at qemu/qemu_domain.c:906 #3 0x00007f5f3c2da7d4 in qemuDomainDestroyFlags (dom=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:1970 #4 0x0000003e026ee111 in virDomainDestroyFlags ( domain=domain@entry=0x7f59dfc6f220, flags=1) at libvirt.c:2264 #5 0x000000000041502d in remoteDispatchDomainDestroyFlags ( args=0x7f59dfc6f260, rerr=0x7f5f439e1c70, client=<optimized out>, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1329 #6 remoteDispatchDomainDestroyFlagsHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f5f439e1c70, args=0x7f59dfc6f260, ret=<optimized out>) at remote_dispatch.h:1307 #7 0x0000003e0274e632 in virNetServerProgramDispatchCall (msg=0xf1f6f0, client=0xf23c40, server=0xefb1e0, prog=0xf1cbb0) at rpc/virnetserverprogram.c:431 #8 virNetServerProgramDispatch (prog=0xf1cbb0, server=server@entry=0xefb1e0, client=0xf23c40, msg=0xf1f6f0) at rpc/virnetserverprogram.c:304 #9 0x0000003e0274a761 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0xefb1e0) at rpc/virnetserver.c:170 #10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0xefb1e0) at rpc/virnetserver.c:191 #11 0x0000003e0266fafe in virThreadPoolWorker (opaque=opaque@entry=0xee9d20) at util/threadpool.c:144 #12 0x0000003e0266f589 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #13 0x0000003578407d15 in start_thread (arg=0x7f5f439e2700) at pthread_create.c:308 #14 0x00000035778f22cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
BTW: (1) I had set NLDBG=4 because I was trying to reproduce another bug. (2) I have libvirtd debugging enabled: log_level=1 log_outputs="1:file:/tmp/libvirtd.log"
Rich, since you had debugging enabled, could you attach the log? And that applies for the other bug as well.
Unfortunately I've overwritten the log now.