Description of problem: I have a setup where i am running ovirt with glusterfs as storagedomain and the domains are accessed via libgfapi. I see that there are lot of core dumps generated by libvirtd. Version-Release number of selected component (if applicable): libvirt-client-1.2.17-13.el7_2.5.x86_64 libvirt-python-1.2.17-2.el7.x86_64 libvirt-lock-sanlock-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64 How reproducible: Steps to Reproduce: 1. Installed HC stack on RHEL7.2 plus ovirt master 2. storage domains are accessed via libgfapi. 3. Actual results: Lot of core dumps in the system generated by livirtd Expected results: libvirtd should not generate any core dumps. Additional info:
copied all the core dumps into the location below: ======================================================== http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/1387558/
Could you analyze at least one of the cores and provide full backtrace (run "thread apply all backtrace" command in gdb)?
Hi, can you please attach a backtrace from the crash. Thanks.
Hi, I tried to analyze core from the bt. But looks like i am missing some debug info packages. Could you please let me know what debug info packages i need to install so that i can provide the bt?
Please also provide the version of libgfapi used. For analyzing the core file, gdb prints a command that can be used to install all the required debuginfo packages using the debuginfo-install tool.
Below is the bt from the core: ================================= (gdb) thread apply all backtrace Thread 41 (Thread 0x7f9204b14700 (LWP 9710)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f9204313700 (LWP 9711)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f91bcff1700 (LWP 9974)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f9207b1a700 (LWP 9704)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269b20) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f9205315700 (LWP 9709)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f91fdad9700 (LWP 9715)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a268100) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f9203b12700 (LWP 9712)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f9207319700 (LWP 9705)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f91fcad7700 (LWP 9717)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a267fa0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f91bfff7700 (LWP 9976)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f91fe2da700 (LWP 9714)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a2681b0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 30 (Thread 0x7f91feadb700 (LWP 9713)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a268260) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f9206317700 (LWP 9707)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f91f8f7f700 (LWP 9763)): #0 0x00007f9200668adf in mem_pool_new_fn () from /lib64/libglusterfs.so.0 #1 0x00007f92009129be in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0 #2 0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0 #3 0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f426ee70) at storage/storage_backend_gluster.c:611 #4 0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f426ee70, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729 #5 0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f426ee70, parent=parent@entry=0x7f91f426ee70, uid=uid@entry=107, gid=gid@entry=107, allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91e0001730) at storage/storage_driver.c:2989 #6 0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f426ee70, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false) at storage/storage_driver.c:3112 #7 0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f42a5190, disk=0x7f91f426ebd0, force_probe=force_probe@entry=true, report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960 #8 0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801 #9 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #10 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f91fd2d8700 (LWP 9716)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a268050) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f9205b16700 (LWP 9708)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f91fa782700 (LWP 9760)): #0 0x00007f9214a52259 in __memset_sse2 () from /lib64/libc.so.6 #1 0x00007f9214a48e3a in calloc () from /lib64/libc.so.6 #2 0x00007f9200668268 in __gf_calloc () from /lib64/libglusterfs.so.0 #3 0x00007f9200668aa7 in mem_pool_new_fn () from /lib64/libglusterfs.so.0 #4 0x00007f9200912a04 in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0 #5 0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0 #6 0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f4165f20) at storage/storage_backend_gluster.c:611 #7 0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f4165f20, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729 #8 0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f4165f20, parent=parent@entry=0x7f91f4165f20, uid=uid@entry=107, gid=gid@entry=107, allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91ec001710) at storage/storage_driver.c:2989 #9 0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f4165f20, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false) at storage/storage_driver.c:3112 #10 0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f4264890, disk=0x7f91f4165c80, force_probe=force_probe@entry=true, report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960 #11 0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801 #12 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #13 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f91bd7f2700 (LWP 9977)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f920831b700 (LWP 9703)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f9206b18700 (LWP 9706)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154 #2 0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a2699c0) at util/virthreadpool.c:105 #3 0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f91c27fc700 (LWP 9973)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f92183728c0 (LWP 9702)): #0 0x00007f9214d97f4d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f9214d93d02 in _L_lock_791 () from /lib64/libpthread.so.0 #2 0x00007f9214d93c08 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f92177282e5 in virMutexLock (m=<optimized out>) at util/virthread.c:89 #4 0x00007f921770f60e in virObjectLock (anyobj=<optimized out>) at util/virobject.c:323 #5 0x00007f91feb5741a in qemuProcessHandleAgentEOF (agent=0x7f91e0001150, vm=0x7f91f42a5190) at qemu/qemu_process.c:139 #6 0x00007f91feb12825 in qemuAgentIO (watch=watch@entry=19, fd=<optimized out>, events=<optimized out>, events@entry=10, opaque=opaque@entry=0x7f91e0001150) at qemu/qemu_agent.c:703 #7 0x00007f92176e6d37 in virEventPollDispatchHandles (fds=<optimized out>, nfds=<optimized out>) at util/vireventpoll.c:509 #8 virEventPollRunOnce () at util/vireventpoll.c:658 #9 0x00007f92176e5402 in virEventRunDefaultImpl () at util/virevent.c:308 #10 0x00007f921782d8d5 in virNetDaemonRun (dmn=0x7f921a276910) at rpc/virnetdaemon.c:701 #11 0x00007f92183f3524 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1577 Thread 19 (Thread 0x7f91bf7f6700 (LWP 9970)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f91785f2700 (LWP 9910)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 17 (Thread 0x7f91beff5700 (LWP 9971)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f91787f4700 (LWP 9904)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 15 (Thread 0x7f91dc724700 (LWP 9860)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 14 (Thread 0x7f91c1ffb700 (LWP 9972)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f91fb4e5700 (LWP 9765)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f91784f1700 (LWP 9913)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 11 (Thread 0x7f91e71fe700 (LWP 9766)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f91dc825700 (LWP 9772)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x00007f91dcc616f3 in ?? () #2 0x0000000000000000 in ?? () Thread 9 (Thread 0x7f91786f3700 (LWP 9905)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 8 (Thread 0x7f91c07f8700 (LWP 9975)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f91d32ff700 (LWP 9861)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 6 (Thread 0x7f91b717a700 (LWP 9863)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 5 (Thread 0x7f91f9780700 (LWP 9762)): #0 0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92177283c6 in virCondWait (c=c@entry=0x7f91e8000fe8, m=m@entry=0x7f91e8000fc0) at util/virthread.c:154 #2 0x00007f91feb7429b in qemuMonitorSend (mon=mon@entry=0x7f91e8000fb0, msg=msg@entry=0x7f91f977fa50) at qemu/qemu_monitor.c:1036 #3 0x00007f91feb88710 in qemuMonitorJSONCommandWithFd (mon=mon@entry=0x7f91e8000fb0, cmd=cmd@entry=0x7f91e83c9930, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7f91f977fae8) at qemu/qemu_monitor_json.c:293 #4 0x00007f91feb8b22e in qemuMonitorJSONCommand (reply=0x7f91f977fae8, cmd=0x7f91e83c9930, mon=0x7f91e8000fb0) at qemu/qemu_monitor_json.c:323 #5 qemuMonitorJSONGetBlockInfo (mon=0x7f91e8000fb0, table=0x7f91e8399740) at qemu/qemu_monitor_json.c:1677 #6 0x00007f91feb7649d in qemuMonitorGetBlockInfo (mon=0x7f91e8000fb0) at qemu/qemu_monitor.c:1766 #7 0x00007f91feb4adfc in qemuDomainCheckEjectableMedia (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f42a60d0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE) at qemu/qemu_hotplug.c:280 #8 0x00007f91feb5f1cf in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3865 #9 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #10 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f91d01d0700 (LWP 9862)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x00007f91b75b66f3 in ?? () #2 0x0000000000000000 in ?? () Thread 3 (Thread 0x7f91dfb58700 (LWP 9829)): #0 0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f91dc06c6f3 in ?? () #2 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f91e7fff700 (LWP 9764)): #0 0x00007f9214a5224f in __memset_sse2 () from /lib64/libc.so.6 #1 0x00007f9214a48e3a in calloc () from /lib64/libc.so.6 #2 0x00007f9200668268 in __gf_calloc () from /lib64/libglusterfs.so.0 #3 0x00007f9200668aa7 in mem_pool_new_fn () from /lib64/libglusterfs.so.0 #4 0x00007f9200912a04 in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0 #5 0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0 #6 0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f415eed0) at storage/storage_backend_gluster.c:611 #7 0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f415eed0, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729 #8 0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f415eed0, parent=parent@entry=0x7f91f415eed0, uid=uid@entry=107, gid=gid@entry=107, allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91d8001fc0) at storage/storage_driver.c:2989 #9 0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f415eed0, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false) at storage/storage_driver.c:3112 #10 0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f42a47d0, disk=0x7f91f415f040, force_probe=force_probe@entry=true, report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960 #11 0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801 #12 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #13 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f9214abeced in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f91f9f81700 (LWP 9761)): #0 0x00007f9200668b3e in mem_pool_new_fn () from /lib64/libglusterfs.so.0 #1 0x00007f92009129be in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0 #2 0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0 #3 0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f4271260) at storage/storage_backend_gluster.c:611 #4 0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f4271260, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729 #5 0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f4271260, parent=parent@entry=0x7f91f4271260, uid=uid@entry=107, gid=gid@entry=107, allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91f00022c0) at storage/storage_driver.c:2989 #6 0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f4271260, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false) at storage/storage_driver.c:3112 #7 0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f4173de0, disk=0x7f91f42714e0, force_probe=force_probe@entry=true, report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960 #8 0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801 #9 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #10 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #11 0x00007f9214abeced in clone () from /lib64/libc.so.6
Looks like it crashed in libgfapi. Reassigning to the glusterfs component since libgfapi seems to be part of glusterfs-api package.
RHGS version ============ glusterfs-client-xlators-3.8.4-2.el7rhgs.x86_64 glusterfs-fuse-3.8.4-2.el7rhgs.x86_64 glusterfs-3.8.4-2.el7rhgs.x86_64 glusterfs-api-3.8.4-2.el7rhgs.x86_64 glusterfs-server-3.8.4-2.el7rhgs.x86_64 glusterfs-libs-3.8.4-2.el7rhgs.x86_64 glusterfs-cli-3.8.4-2.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-2.el7rhgs.x86_64
Patch posted uptream: http://review.gluster.org/15842
Upstream master : http://review.gluster.org/15842 Upstream release 3.7: http://review.gluster.org/15910 Upstream release 3.8: http://review.gluster.org/15909 Upstream release 3.9: http://review.gluster.org/15908 Downstream : https://code.engineering.redhat.com/gerrit/90965
Verified with build glusterfs-3.8.4-10.el7rhgs.x86_64. Followed the blog[1] to store vm images in the gluster volume and access them via libgfapi. [1] https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md. Did not see any core dumps in the system, so moving this to verified state. will reopen this bug if hit again.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html