Bug 1387558

Summary: libgfapi core dumps
Product: Red Hat Gluster Storage Reporter: RamaKasturi <knarra>
Component: libgfapiAssignee: rjoseph
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: amukherj, dyuan, jdenemar, knarra, pkrempa, rbalakri, rhinduja, rhs-bugs, sasundar, storage-qa-internal, xuzhang, yafu, zpeng
Target Milestone: ---   
Target Release: RHGS 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1394719 (view as bug list) Environment:
Last Closed: 2017-03-23 06:14:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1351528, 1394719, 1397662, 1397663, 1397664    

Description RamaKasturi 2016-10-21 09:06:25 UTC
Description of problem:
I have a setup where i am running ovirt with glusterfs as storagedomain and the domains are accessed via libgfapi. I see that there are lot of core dumps generated by libvirtd.

Version-Release number of selected component (if applicable):
libvirt-client-1.2.17-13.el7_2.5.x86_64
libvirt-python-1.2.17-2.el7.x86_64
libvirt-lock-sanlock-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64


How reproducible:


Steps to Reproduce:
1. Installed HC stack on RHEL7.2 plus ovirt master
2. storage domains are accessed via libgfapi.
3.

Actual results:
Lot of core dumps in the system generated by livirtd

Expected results:
libvirtd should not generate any core dumps.

Additional info:

Comment 2 RamaKasturi 2016-10-21 09:24:00 UTC
copied all the core dumps into the location below:
========================================================
http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/1387558/

Comment 3 Jiri Denemark 2016-10-21 09:39:35 UTC
Could you analyze at least one of the cores and provide full backtrace (run "thread apply all backtrace" command in gdb)?

Comment 4 Jaroslav Suchanek 2016-10-21 09:39:37 UTC
Hi, can you please attach a backtrace from the crash. Thanks.

Comment 5 RamaKasturi 2016-10-21 09:48:14 UTC
Hi, 
 
   I tried to analyze core from the bt. But looks like i am missing some debug info packages. Could you please let me know what debug info packages i need to install so that i can provide the bt?

Comment 6 Peter Krempa 2016-10-21 10:26:46 UTC
Please also provide the version of libgfapi used.

For analyzing the core file, gdb prints a command that can be used to install all the required debuginfo packages using the debuginfo-install tool.

Comment 7 RamaKasturi 2016-10-21 12:37:41 UTC
Below is the bt from the core:
=================================
(gdb) thread apply all backtrace

Thread 41 (Thread 0x7f9204b14700 (LWP 9710)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f9204313700 (LWP 9711)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f91bcff1700 (LWP 9974)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f9207b1a700 (LWP 9704)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269b20) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f9205315700 (LWP 9709)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f91fdad9700 (LWP 9715)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit--- 
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a268100) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f9203b12700 (LWP 9712)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f9207319700 (LWP 9705)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f91fcad7700 (LWP 9717)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a267fa0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f91bfff7700 (LWP 9976)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f91fe2da700 (LWP 9714)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a2681b0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 30 (Thread 0x7f91feadb700 (LWP 9713)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a268260) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f9206317700 (LWP 9707)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f91f8f7f700 (LWP 9763)):
#0  0x00007f9200668adf in mem_pool_new_fn () from /lib64/libglusterfs.so.0
#1  0x00007f92009129be in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0
#2  0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0
#3  0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f426ee70) at storage/storage_backend_gluster.c:611
#4  0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f426ee70, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729
#5  0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f426ee70, parent=parent@entry=0x7f91f426ee70, uid=uid@entry=107, gid=gid@entry=107, 
    allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91e0001730) at storage/storage_driver.c:2989
#6  0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f426ee70, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false)
    at storage/storage_driver.c:3112
#7  0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f42a5190, disk=0x7f91f426ebd0, force_probe=force_probe@entry=true, 
    report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960
#8  0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801
#9  0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#10 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f91fd2d8700 (LWP 9716)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a287668, m=m@entry=0x7f921a287640) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a268050) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f9205b16700 (LWP 9708)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274d50, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ceb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269a70) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f91fa782700 (LWP 9760)):
#0  0x00007f9214a52259 in __memset_sse2 () from /lib64/libc.so.6
#1  0x00007f9214a48e3a in calloc () from /lib64/libc.so.6
#2  0x00007f9200668268 in __gf_calloc () from /lib64/libglusterfs.so.0
#3  0x00007f9200668aa7 in mem_pool_new_fn () from /lib64/libglusterfs.so.0
#4  0x00007f9200912a04 in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0
#5  0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0
#6  0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f4165f20) at storage/storage_backend_gluster.c:611
#7  0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f4165f20, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729
#8  0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f4165f20, parent=parent@entry=0x7f91f4165f20, uid=uid@entry=107, gid=gid@entry=107, 
    allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91ec001710) at storage/storage_driver.c:2989
#9  0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f4165f20, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false)
    at storage/storage_driver.c:3112
#10 0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f4264890, disk=0x7f91f4165c80, force_probe=force_probe@entry=true, 
    report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960
#11 0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801
#12 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#13 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f91bd7f2700 (LWP 9977)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f920831b700 (LWP 9703)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a269bd0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f9206b18700 (LWP 9706)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f921a274cb8, m=m@entry=0x7f921a274c90) at util/virthread.c:154
#2  0x00007f9217728ccb in virThreadPoolWorker (opaque=opaque@entry=0x7f921a2699c0) at util/virthreadpool.c:105
#3  0x00007f9217728158 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f91c27fc700 (LWP 9973)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f92183728c0 (LWP 9702)):
#0  0x00007f9214d97f4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9214d93d02 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007f9214d93c08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f92177282e5 in virMutexLock (m=<optimized out>) at util/virthread.c:89
#4  0x00007f921770f60e in virObjectLock (anyobj=<optimized out>) at util/virobject.c:323
#5  0x00007f91feb5741a in qemuProcessHandleAgentEOF (agent=0x7f91e0001150, vm=0x7f91f42a5190) at qemu/qemu_process.c:139
#6  0x00007f91feb12825 in qemuAgentIO (watch=watch@entry=19, fd=<optimized out>, events=<optimized out>, events@entry=10, opaque=opaque@entry=0x7f91e0001150)
    at qemu/qemu_agent.c:703
#7  0x00007f92176e6d37 in virEventPollDispatchHandles (fds=<optimized out>, nfds=<optimized out>) at util/vireventpoll.c:509
#8  virEventPollRunOnce () at util/vireventpoll.c:658
#9  0x00007f92176e5402 in virEventRunDefaultImpl () at util/virevent.c:308
#10 0x00007f921782d8d5 in virNetDaemonRun (dmn=0x7f921a276910) at rpc/virnetdaemon.c:701
#11 0x00007f92183f3524 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1577

Thread 19 (Thread 0x7f91bf7f6700 (LWP 9970)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f91785f2700 (LWP 9910)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f91beff5700 (LWP 9971)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f91787f4700 (LWP 9904)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f91dc724700 (LWP 9860)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f91c1ffb700 (LWP 9972)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f91fb4e5700 (LWP 9765)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f91784f1700 (LWP 9913)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f91e71fe700 (LWP 9766)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f91dc825700 (LWP 9772)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007f91dcc616f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f91786f3700 (LWP 9905)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f91c07f8700 (LWP 9975)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f920067ab08 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007f920067b950 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f91d32ff700 (LWP 9861)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f91b717a700 (LWP 9863)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f91f9780700 (LWP 9762)):
#0  0x00007f9214d956d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f92177283c6 in virCondWait (c=c@entry=0x7f91e8000fe8, m=m@entry=0x7f91e8000fc0) at util/virthread.c:154
#2  0x00007f91feb7429b in qemuMonitorSend (mon=mon@entry=0x7f91e8000fb0, msg=msg@entry=0x7f91f977fa50) at qemu/qemu_monitor.c:1036
#3  0x00007f91feb88710 in qemuMonitorJSONCommandWithFd (mon=mon@entry=0x7f91e8000fb0, cmd=cmd@entry=0x7f91e83c9930, scm_fd=scm_fd@entry=-1, 
    reply=reply@entry=0x7f91f977fae8) at qemu/qemu_monitor_json.c:293
#4  0x00007f91feb8b22e in qemuMonitorJSONCommand (reply=0x7f91f977fae8, cmd=0x7f91e83c9930, mon=0x7f91e8000fb0) at qemu/qemu_monitor_json.c:323
#5  qemuMonitorJSONGetBlockInfo (mon=0x7f91e8000fb0, table=0x7f91e8399740) at qemu/qemu_monitor_json.c:1677
#6  0x00007f91feb7649d in qemuMonitorGetBlockInfo (mon=0x7f91e8000fb0) at qemu/qemu_monitor.c:1766
#7  0x00007f91feb4adfc in qemuDomainCheckEjectableMedia (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f42a60d0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE)
    at qemu/qemu_hotplug.c:280
#8  0x00007f91feb5f1cf in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3865
#9  0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#10 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f91d01d0700 (LWP 9862)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007f91b75b66f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f91dfb58700 (LWP 9829)):
#0  0x00007f9214d95a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91dc06c6f3 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f91e7fff700 (LWP 9764)):
#0  0x00007f9214a5224f in __memset_sse2 () from /lib64/libc.so.6
#1  0x00007f9214a48e3a in calloc () from /lib64/libc.so.6
#2  0x00007f9200668268 in __gf_calloc () from /lib64/libglusterfs.so.0
#3  0x00007f9200668aa7 in mem_pool_new_fn () from /lib64/libglusterfs.so.0
#4  0x00007f9200912a04 in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0
#5  0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0
#6  0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f415eed0) at storage/storage_backend_gluster.c:611
#7  0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f415eed0, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729
#8  0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f415eed0, parent=parent@entry=0x7f91f415eed0, uid=uid@entry=107, gid=gid@entry=107, 
    allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91d8001fc0) at storage/storage_driver.c:2989
#9  0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f415eed0, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false)
    at storage/storage_driver.c:3112
#10 0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f42a47d0, disk=0x7f91f415f040, force_probe=force_probe@entry=true, 
    report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960
#11 0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801
#12 0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#13 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f9214abeced in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f91f9f81700 (LWP 9761)):
#0  0x00007f9200668b3e in mem_pool_new_fn () from /lib64/libglusterfs.so.0
#1  0x00007f92009129be in glusterfs_ctx_defaults_init () from /lib64/libgfapi.so.0
#2  0x00007f92009138c4 in glfs_new () from /lib64/libgfapi.so.0
#3  0x00007f92021f1e2e in virStorageFileBackendGlusterInit (src=0x7f91f4271260) at storage/storage_backend_gluster.c:611
#4  0x00007f92021e0202 in virStorageFileInitAs (src=src@entry=0x7f91f4271260, uid=uid@entry=107, gid=gid@entry=107) at storage/storage_driver.c:2729
#5  0x00007f92021e07c3 in virStorageFileGetMetadataRecurse (src=src@entry=0x7f91f4271260, parent=parent@entry=0x7f91f4271260, uid=uid@entry=107, gid=gid@entry=107, 
    allow_probe=allow_probe@entry=false, report_broken=report_broken@entry=false, cycle=cycle@entry=0x7f91f00022c0) at storage/storage_driver.c:2989
#6  0x00007f92021e0cbf in virStorageFileGetMetadata (src=0x7f91f4271260, uid=107, gid=107, allow_probe=false, report_broken=report_broken@entry=false)
    at storage/storage_driver.c:3112
#7  0x00007f91feb45185 in qemuDomainDetermineDiskChain (driver=driver@entry=0x7f91f40f0de0, vm=0x7f91f4173de0, disk=0x7f91f42714e0, force_probe=force_probe@entry=true, 
    report_broken=report_broken@entry=false) at qemu/qemu_domain.c:2960
#8  0x00007f91feb5eb74 in qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:3801
#9  0x00007f9217728182 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#10 0x00007f9214d91dc5 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#11 0x00007f9214abeced in clone () from /lib64/libc.so.6

Comment 8 Peter Krempa 2016-10-21 12:58:18 UTC
Looks like it crashed in libgfapi. Reassigning to the glusterfs component since libgfapi seems to be part of glusterfs-api package.

Comment 10 RamaKasturi 2016-10-25 10:09:56 UTC
RHGS version
============
glusterfs-client-xlators-3.8.4-2.el7rhgs.x86_64
glusterfs-fuse-3.8.4-2.el7rhgs.x86_64
glusterfs-3.8.4-2.el7rhgs.x86_64
glusterfs-api-3.8.4-2.el7rhgs.x86_64
glusterfs-server-3.8.4-2.el7rhgs.x86_64
glusterfs-libs-3.8.4-2.el7rhgs.x86_64
glusterfs-cli-3.8.4-2.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-2.el7rhgs.x86_64

Comment 11 rjoseph 2016-11-15 06:14:08 UTC
Patch posted uptream: http://review.gluster.org/15842

Comment 14 rjoseph 2016-11-23 06:53:15 UTC
Upstream master     : http://review.gluster.org/15842
Upstream release 3.7: http://review.gluster.org/15910
Upstream release 3.8: http://review.gluster.org/15909
Upstream release 3.9: http://review.gluster.org/15908

Downstream : https://code.engineering.redhat.com/gerrit/90965

Comment 16 RamaKasturi 2016-12-29 12:02:52 UTC
Verified with build glusterfs-3.8.4-10.el7rhgs.x86_64. 

Followed the blog[1] to store vm images in the gluster volume and access them via libgfapi. 

[1] https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md.

Did not see any core dumps in the system, so moving this to verified state.
will reopen this bug if  hit  again.

Comment 18 errata-xmlrpc 2017-03-23 06:14:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html