Bug 1703434

Summary: Post inservice upgrade of first node in the cluster, core generated and heal on EC stuck
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Kshithij Iyer <kiyer>
Component: disperseAssignee: Mohammed Rafi KC <rkavunga>
Status: CLOSED DEFERRED QA Contact: Kshithij Iyer <kiyer>
Severity: urgent Docs Contact:
Priority: high    
Version: rhgs-3.5CC: amukherj, aspandey, nchilaka, rhinduja, rhs-bugs, rkavunga, sheggodu, srakonde, storage-qa-internal
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: shd-multiplexing
Fixed In Version: glusterfs-6.0-5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-20 09:31:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1713664    
Bug Blocks:    

Description Kshithij Iyer 2019-04-26 12:12:00 UTC 
Description of problem:
On a 6 node gluster cluster created one volume of each type(Replicated, Distributed Replicated, Disperse, Distributed Disperse, Arbiter and Distributed Arbiter), mounted all the volumes to clients and started IO and then started upgrading nodes one by one from 3.4.4(3.12.2-47) to 3.5.0(glusterfs-6.0-2) after upgrading the first node observed that heal was stuck for about 30 minutes and a core file was generated.

################################################################################
Just after upgrade
################################################################################
# for volume in `gluster v list`; do echo $volume;gluster v heal $volume info | grep "Number of entries:";done
arb
Number of entries: 0
Number of entries: 1377
Number of entries: 1199
disp
Number of entries: 0
Number of entries: 403
Number of entries: 287
Number of entries: 200
Number of entries: 124
Number of entries: 58
dist
distarb
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
distdisp
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 19
Number of entries: 19
Number of entries: 19
Number of entries: 19
Number of entries: 19
distrep
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
rep
Number of entries: 0
Number of entries: 642
Number of entries: 427
################################################################################
After about 30 minutes:
################################################################################
# for volume in `gluster v list`; do echo $volume;gluster v heal $volume info | grep "Number of entries:";done
arb
Number of entries: 0
Number of entries: 0
Number of entries: 0
disp
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
dist
distarb
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
distdisp
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 19
Number of entries: 19
Number of entries: 19
Number of entries: 19
Number of entries: 19
distrep
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
rep
Number of entries: 0
Number of entries: 0
Number of entries: 0
################################################################################
Program terminated with signal 11, Segmentation fault.
#0  ec_shd_index_heal (subvol=0x7f2ca804ae50, 
    entry=0x7f2ca4125040, parent=0x7f2c65ffae10, 
    data=0x7f2ca82ee5a0) at ec-heald.c:183
183	    if (ec->xl_up_count <= ec->fragments)
################################################################################
(gdb) bt 
#0  ec_shd_index_heal (subvol=0x7f2ca804ae50, entry=0x7f2ca4125040, parent=0x7f2c65ffae10, data=0x7f2ca82ee5a0) at ec-heald.c:183
#1  0x00007f2cc1145c72 in _dir_scan_job_fn (data=0x7f2c440045a0) at syncop-utils.c:264
#2  0x00007f2cc111e470 in synctask_wrap () at syncop.c:367
#3  0x00007f2cbf72f010 in ?? () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
################################################################################
(gdb) t a a bt 

Thread 64 (Thread 0x7f2c67fff700 (LWP 6977)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f5070) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f5070) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7f2c2f7fe700 (LWP 7167)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f5bb0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f5bb0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7f2cb6eb9700 (LWP 5928)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f4170) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f4170) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7f2cb76ba700 (LWP 5927)):
#0  0x00007f2cbf7abe2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f2cbf7abcc4 in sleep () from /lib64/libc.so.6
#2  0x00007f2cc110b5bd in pool_sweeper (arg=<optimized out>) at mem-pool.c:473
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7f2c437fe700 (LWP 7088)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca8008ec0, loc=loc@entry=0x7f2c437fde10, pid=pid@entry=-6, data=data@entry=0x7f2ca8027e90, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c30001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca8027e90) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca8027e90) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 59 (Thread 0x7f2c127f4700 (LWP 9828)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc111c19b in glusterfs_graph_cleanup (arg=0x7f2c3039a200) at graph.c:1372
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7f2c2cff9700 (LWP 7302)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f6e70) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f6e70) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7f2cb33ee700 (LWP 5936)):
#0  0x00007f2cbf7e5483 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f2cc1145050 in event_dispatch_epoll_worker (data=0x559e7ea3ded0) at event-epoll.c:751
---Type <return> to continue, or q <return> to quit---
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7f2c157fa700 (LWP 9656)):
#0  0x00007f2cbff244ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2cbff1fdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f2cbff1fc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f2cc10e0bff in _gf_log_callingfn (domain=0x7f2ca8310020 "distdisp-client-2", file=file@entry=0x7f2cc0e8fae2 "rpc-clnt.c", 
    function=function@entry=0x7f2cc0e8fea0 <__FUNCTION__.17981> "saved_frames_unwind", line=line@entry=346, level=level@entry=GF_LOG_ERROR, 
    fmt=fmt@entry=0x7f2cc0e8f4f8 "forced unwinding frame type(%s) op(%s(%d)) called at %s (xid=0x%x)") at logging.c:937
#4  0x00007f2cc0e86744 in saved_frames_unwind (saved_frames=saved_frames@entry=0x7f2cac4581f0) at rpc-clnt.c:338
#5  0x00007f2cc0e8685e in saved_frames_destroy (frames=0x7f2cac4581f0) at rpc-clnt.c:366
#6  0x00007f2cc0e87913 in rpc_clnt_connection_cleanup (conn=0x7f2ca8313100) at rpc-clnt.c:536
#7  0x00007f2cb295e603 in fini (this=<optimized out>) at client.c:2788
#8  0x00007f2cc111c112 in glusterfs_graph_fini (graph=graph@entry=0x7f2ca80341e0) at graph.c:1195
#9  0x00007f2cc111c206 in glusterfs_graph_cleanup (arg=0x7f2ca80341e0) at graph.c:1395
#10 0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7f2c147f8700 (LWP 9687)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f79b0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f79b0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7f2c0dfeb700 (LWP 9895)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb2710ebe in __afr_shd_healer_wait (healer=healer@entry=0x7f2c9c2b7f60) at afr-self-heald.c:100
#2  0x00007f2cb2710f3d in afr_shd_healer_wait (healer=healer@entry=0x7f2c9c2b7f60) at afr-self-heald.c:121
#3  0x00007f2cb2712a20 in afr_shd_index_healer (data=0x7f2c9c2b7f60) at afr-self-heald.c:820
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7f2c087e0700 (LWP 10273)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2c60db6a90) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2c60db6a90) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2c60db6a90) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7f2c04fd9700 (LWP 10291)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc11465bb in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c419470, loc=loc@entry=0x7f2c04fd8e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c4368d0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c202184a8, max_jobs=1, max_qlen=1024) at syncop-utils.c:428
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c4368d0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c4368d0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7f2cb86bc700 (LWP 5925)):
#0  0x00007f2cbff24e3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f2cc10ee1c6 in gf_timer_proc (data=0x559e7e9f3950) at timer.c:194
---Type <return> to continue, or q <return> to quit---
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7f2c777fe700 (LWP 6944)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1170) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1170) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b1170) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7f2c76ffd700 (LWP 6945)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b11e0) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b11e0) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b11e0) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7f2c677fe700 (LWP 7070)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca82ee450) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca82ee450) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca82ee450) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7f2c66ffd700 (LWP 7071)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca8043ab0, loc=loc@entry=0x7f2c66ffce10, pid=pid@entry=-6, data=data@entry=0x7f2ca82ee4c0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c3c001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca82ee4c0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca82ee4c0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7f2c427fc700 (LWP 7090)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca8010240, loc=loc@entry=0x7f2c427fbe10, pid=pid@entry=-6, data=data@entry=0x7f2ca8027f70, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c34001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca8027f70) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca8027f70) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7f2cb7ebb700 (LWP 5926)):
#0  0x00007f2cbff25361 in sigwait () from /lib64/libpthread.so.0
#1  0x0000559e7d19543b in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2370
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7f2c047d8700 (LWP 10292)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc11465bb in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c41cb90, loc=loc@entry=0x7f2c047d7e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c436940, 
---Type <return> to continue, or q <return> to quit---
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c28218748, max_jobs=1, max_qlen=1024) at syncop-utils.c:428
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c436940) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c436940) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7f2c7f7fe700 (LWP 6686)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f4cb0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f4cb0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7f2c14ff9700 (LWP 9686)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f75f0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f75f0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7f2c2dffb700 (LWP 7300)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f66f0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f66f0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f2c2e7fc700 (LWP 7234)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f6330) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f6330) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f2c167fc700 (LWP 8543)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f7230) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f7230) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f2c2d7fa700 (LWP 7301)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f6ab0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f6ab0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f2c2ffff700 (LWP 7104)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f57f0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f57f0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 36 (Thread 0x7f2c2effd700 (LWP 7168)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f5f70) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f5f70) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f2cb66b8700 (LWP 5929)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f4530) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f4530) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f2c417fa700 (LWP 7092)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca80175c0, loc=loc@entry=0x7f2c417f9e10, pid=pid@entry=-6, data=data@entry=0x7f2ca8028050, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c24001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca8028050) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca8028050) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f2c41ffb700 (LWP 7091)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca8013c00, loc=loc@entry=0x7f2c41ffae10, pid=pid@entry=-6, data=data@entry=0x7f2ca8027fe0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c28001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca8027fe0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca8027fe0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f2c42ffd700 (LWP 7089)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca800c880, loc=loc@entry=0x7f2c42ffce10, pid=pid@entry=-6, data=data@entry=0x7f2ca8027f00, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c20001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca8027f00) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca8027f00) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f2c64ff9700 (LWP 7075)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca80521a0, loc=loc@entry=0x7f2c64ff8e10, pid=pid@entry=-6, data=data@entry=0x7f2ca82ee680, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c58001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca82ee680) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca82ee680) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f2c657fa700 (LWP 7074)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca804e830, loc=loc@entry=0x7f2c657f9e10, pid=pid@entry=-6, data=data@entry=0x7f2ca82ee610, 
---Type <return> to continue, or q <return> to quit---
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c50001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca82ee610) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca82ee610) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f2c65ffb700 (LWP 7073)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca804ae50, loc=loc@entry=0x7f2c65ffae10, pid=pid@entry=-6, data=data@entry=0x7f2ca82ee5a0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c44001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca82ee5a0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca82ee5a0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f2c667fc700 (LWP 7072)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2ca8047470, loc=loc@entry=0x7f2c667fbe10, pid=pid@entry=-6, data=data@entry=0x7f2ca82ee530, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c4c001cd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2ca82ee530) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2ca82ee530) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f2c74ff9700 (LWP 6949)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b13a0) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b13a0) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b13a0) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f2c75ffb700 (LWP 6947)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b12c0) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b12c0) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b12c0) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f2c767fc700 (LWP 6946)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1250) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1250) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b1250) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f2cc15de780 (LWP 5923)):
#0  0x00007f2cbff1ef47 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007f2cc1144468 in event_dispatch_epoll (event_pool=0x559e7e9eb5b0) at event-epoll.c:846
#2  0x0000559e7d1919b5 in main (argc=16, argv=<optimized out>) at glusterfsd.c:2866

Thread 23 (Thread 0x7f2c02fd5700 (LWP 10295)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc11465bb in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c427240, loc=loc@entry=0x7f2c02fd4e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c436a90, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c30398508, max_jobs=1, max_qlen=1024) at syncop-utils.c:428
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c436a90) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c436a90) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f2c03fd7700 (LWP 10293)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc11465bb in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c420320, loc=loc@entry=0x7f2c03fd6e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c4369b0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c3420d178, max_jobs=1, max_qlen=1024) at syncop-utils.c:428
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c4369b0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c4369b0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f2c057da700 (LWP 10290)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c417ac0, loc=loc@entry=0x7f2c057d9e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c436860, 
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c79630) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c79630) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f2c13ff7700 (LWP 9988)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb2710ebe in __afr_shd_healer_wait (healer=healer@entry=0x7f2c5c339740) at afr-self-heald.c:100
#2  0x00007f2cb2710f3d in afr_shd_healer_wait (healer=healer@entry=0x7f2c5c339740) at afr-self-heald.c:121
#3  0x00007f2cb2712a20 in afr_shd_index_healer (data=0x7f2c5c339740) at afr-self-heald.c:820
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f2c15ffb700 (LWP 9642)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc111c19b in glusterfs_graph_cleanup (arg=0x7f2ca8004f50) at graph.c:1372
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f2c7ffff700 (LWP 6681)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f48f0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f48f0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f2c757fa700 (LWP 6948)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1330) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1330) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b1330) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f2c43fff700 (LWP 7087)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca8027e20) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca8027e20) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca8027e20) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f2cb5eb7700 (LWP 5930)):
#0  0x00007f2cbf7dbf73 in select () from /lib64/libc.so.6
#1  0x00007f2cc115f7f4 in runner (arg=0x559e7e9f82d0) at ../../contrib/timer-wheel/timer-wheel.c:186
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f2c037d6700 (LWP 10294)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc11465bb in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c423ab0, loc=loc@entry=0x7f2c037d5e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c436a20, 
---Type <return> to continue, or q <return> to quit---
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c3c6732f8, max_jobs=1, max_qlen=1024) at syncop-utils.c:428
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c436a20) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c436a20) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f2c05fdb700 (LWP 10278)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6021a590, loc=loc@entry=0x7f2c05fdae10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6cc0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c24239738, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6cc0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6cc0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f2c067dc700 (LWP 10277)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60216c80, loc=loc@entry=0x7f2c067dbe10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6c50, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c18621c28, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6c50) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6c50) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f2c06fdd700 (LWP 10276)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60213210, loc=loc@entry=0x7f2c06fdce10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6be0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c1c4042b8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6be0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6be0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f2c077de700 (LWP 10275)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6020f8a0, loc=loc@entry=0x7f2c077dde10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6b70, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x559e7ee7cc68, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6b70) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6b70) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f2c07fdf700 (LWP 10274)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6020bed0, loc=loc@entry=0x7f2c07fdee10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6b00, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2cac097268, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6b00) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6b00) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f2c097e2700 (LWP 10175)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6022c820, loc=loc@entry=0x7f2c097e1e10, pid=pid@entry=-6, data=data@entry=0x7f2c60c79780, 
---Type <return> to continue, or q <return> to quit---
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c9c5b12c8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c79780) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c79780) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f2c12ff5700 (LWP 10171)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2c60c795c0) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2c60c795c0) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2c60c795c0) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f2cb3bef700 (LWP 5935)):
#0  0x00007f2cbf7e5483 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f2cc1145050 in event_dispatch_epoll_worker (data=0x559e7ea3c0d0) at event-epoll.c:751
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f2c40ff9700 (LWP 7103)):
#0  ec_shd_index_heal (subvol=0x7f2ca804ae50, entry=0x7f2ca4125040, parent=0x7f2c65ffae10, data=0x7f2ca82ee5a0) at ec-heald.c:183
#1  0x00007f2cc1145c72 in _dir_scan_job_fn (data=0x7f2c440045a0) at syncop-utils.c:264
#2  0x00007f2cc111e470 in synctask_wrap () at syncop.c:367
#3  0x00007f2cbf72f010 in ?? () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c38637dd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c436860) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c436860) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f2c08fe1700 (LWP 10176)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60230190, loc=loc@entry=0x7f2c08fe0e10, pid=pid@entry=-6, data=data@entry=0x7f2c60c797f0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c98272628, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c797f0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c797f0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f2c09fe3700 (LWP 10174)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60228e50, loc=loc@entry=0x7f2c09fe2e10, pid=pid@entry=-6, data=data@entry=0x7f2c60c79710, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c9001e178, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c79710) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c79710) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f2c8effd700 (LWP 10173)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60225390, loc=loc@entry=0x7f2c8effce10, pid=pid@entry=-6, data=data@entry=0x7f2c60c796a0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2ca8a44dd8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c796a0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c796a0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f2c11ff3700 (LWP 10172)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c602219f0, loc=loc@entry=0x7f2c11ff2e10, pid=pid@entry=-6, data=data@entry=0x7f2c60c79630, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c94010318, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c79630) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c79630) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f2c13ff7700 (LWP 9988)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb2710ebe in __afr_shd_healer_wait (healer=healer@entry=0x7f2c5c339740) at afr-self-heald.c:100
#2  0x00007f2cb2710f3d in afr_shd_healer_wait (healer=healer@entry=0x7f2c5c339740) at afr-self-heald.c:121
#3  0x00007f2cb2712a20 in afr_shd_index_healer (data=0x7f2c5c339740) at afr-self-heald.c:820
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f2c15ffb700 (LWP 9642)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc111c19b in glusterfs_graph_cleanup (arg=0x7f2ca8004f50) at graph.c:1372
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f2c7ffff700 (LWP 6681)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1120800 in syncenv_task (proc=proc@entry=0x559e7e9f48f0) at syncop.c:612
#2  0x00007f2cc11216b0 in syncenv_processor (thdata=0x559e7e9f48f0) at syncop.c:679
#3  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f2c757fa700 (LWP 6948)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1330) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca81b1330) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca81b1330) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f2c43fff700 (LWP 7087)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2ca8027e20) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2ca8027e20) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2ca8027e20) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f2cb5eb7700 (LWP 5930)):
#0  0x00007f2cbf7dbf73 in select () from /lib64/libc.so.6
#1  0x00007f2cc115f7f4 in runner (arg=0x559e7e9f82d0) at ../../contrib/timer-wheel/timer-wheel.c:186
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f2c037d6700 (LWP 10294)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc11465bb in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c1c423ab0, loc=loc@entry=0x7f2c037d5e10, pid=pid@entry=-6, data=data@entry=0x7f2c1c436a20, 
---Type <return> to continue, or q <return> to quit---
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c3c6732f8, max_jobs=1, max_qlen=1024) at syncop-utils.c:428
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c1c436a20) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c1c436a20) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f2c05fdb700 (LWP 10278)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6021a590, loc=loc@entry=0x7f2c05fdae10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6cc0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c24239738, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6cc0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6cc0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f2c067dc700 (LWP 10277)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60216c80, loc=loc@entry=0x7f2c067dbe10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6c50, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c18621c28, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6c50) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6c50) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f2c06fdd700 (LWP 10276)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c60213210, loc=loc@entry=0x7f2c06fdce10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6be0, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c1c4042b8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6be0) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6be0) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f2c077de700 (LWP 10275)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6020f8a0, loc=loc@entry=0x7f2c077dde10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6b70, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x559e7ee7cc68, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6b70) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6b70) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f2c07fdf700 (LWP 10274)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6020bed0, loc=loc@entry=0x7f2c07fdee10, pid=pid@entry=-6, data=data@entry=0x7f2c60db6b00, 
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2cac097268, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60db6b00) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60db6b00) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f2c097e2700 (LWP 10175)):
#0  0x00007f2cbff21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cc1146703 in syncop_mt_dir_scan (frame=frame@entry=0x0, subvol=subvol@entry=0x7f2c6022c820, loc=loc@entry=0x7f2c097e1e10, pid=pid@entry=-6, data=data@entry=0x7f2c60c79780, 
---Type <return> to continue, or q <return> to quit---
    fn=fn@entry=0x7f2cb044b320 <ec_shd_index_heal>, xdata=xdata@entry=0x7f2c9c5b12c8, max_jobs=1, max_qlen=1024) at syncop-utils.c:457
#2  0x00007f2cb044b715 in ec_shd_index_sweep (healer=healer@entry=0x7f2c60c79780) at ec-heald.c:244
#3  0x00007f2cb044b8a0 in ec_shd_index_healer (data=0x7f2c60c79780) at ec-heald.c:331
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f2c12ff5700 (LWP 10171)):
#0  0x00007f2cbff21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2cb044b03e in __ec_shd_healer_wait (healer=healer@entry=0x7f2c60c795c0) at ec-heald.c:69
#2  0x00007f2cb044b0ad in ec_shd_healer_wait (healer=healer@entry=0x7f2c60c795c0) at ec-heald.c:90
#3  0x00007f2cb044b83a in ec_shd_index_healer (data=0x7f2c60c795c0) at ec-heald.c:326
#4  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f2cb3bef700 (LWP 5935)):
#0  0x00007f2cbf7e5483 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f2cc1145050 in event_dispatch_epoll_worker (data=0x559e7ea3c0d0) at event-epoll.c:751
#2  0x00007f2cbff1ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f2cbf7e4ead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f2c40ff9700 (LWP 7103)):
#0  ec_shd_index_heal (subvol=0x7f2ca804ae50, entry=0x7f2ca4125040, parent=0x7f2c65ffae10, data=0x7f2ca82ee5a0) at ec-heald.c:183
#1  0x00007f2cc1145c72 in _dir_scan_job_fn (data=0x7f2c440045a0) at syncop-utils.c:264
#2  0x00007f2cc111e470 in synctask_wrap () at syncop.c:367
#3  0x00007f2cbf72f010 in ?? () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
################################################################################################################################################################
# pstack `pidof glusterd`
Thread 9 (Thread 0x7f3e24e42700 (LWP 5194)):
#0  0x00007f3e2c6aae3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f3e2d8741c6 in gf_timer_proc (data=0x55a20b32d480) at timer.c:194
#2  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f3e24641700 (LWP 5195)):
#0  0x00007f3e2c6ab361 in sigwait () from /lib64/libpthread.so.0
#1  0x000055a209f1c43b in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2370
#2  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f3e23e40700 (LWP 5196)):
#0  0x00007f3e2bf31e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3e2bf31cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f3e2d8915bd in pool_sweeper (arg=<optimized out>) at mem-pool.c:473
#3  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f3e2363f700 (LWP 5197)):
#0  0x00007f3e2c6a7d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3e2d8a6800 in syncenv_task (proc=proc@entry=0x55a20b32dc10) at syncop.c:612
#2  0x00007f3e2d8a76b0 in syncenv_processor (thdata=0x55a20b32dc10) at syncop.c:679
#3  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f3e22e3e700 (LWP 5198)):
#0  0x00007f3e2c6a7d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3e2d8a6800 in syncenv_task (proc=proc@entry=0x55a20b32dfd0) at syncop.c:612
#2  0x00007f3e2d8a76b0 in syncenv_processor (thdata=0x55a20b32dfd0) at syncop.c:679
#3  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3e2263d700 (LWP 5199)):
#0  0x00007f3e2bf61f73 in select () from /lib64/libc.so.6
#1  0x00007f3e2d8e57f4 in runner (arg=0x55a20b331d70) at ../../contrib/timer-wheel/timer-wheel.c:186
#2  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3e1ee4d700 (LWP 5503)):
#0  0x00007f3e2c6a7965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3e21a49f2b in hooks_worker (args=<optimized out>) at glusterd-hooks.c:527
#2  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3e1e64c700 (LWP 5504)):
#0  0x00007f3e2bf6b483 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f3e2d8cb050 in event_dispatch_epoll_worker (data=0x55a20b3a5bc0) at event-epoll.c:751
#2  0x00007f3e2c6a3dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3e2bf6aead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3e2dd64780 (LWP 5193)):
#0  0x00007f3e2c6a4f47 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007f3e2d8ca468 in event_dispatch_epoll (event_pool=0x55a20b3255b0) at event-epoll.c:846
#2  0x000055a209f189b5 in main (argc=5, argv=<optimized out>) at glusterfsd.c:2866
################################################################################

Version-Release number of selected component (if applicable):
glusterfs-6.0-2

How reproducible:
1/1

Steps to Reproduce:
1. Create a 6 node cluster.
2. Create volumes of all type.
3. Mount volumes of each type and start IO on each volume.
4. Perform an inservice upgrade to 3.5.0.

Actual results:
Upgrade should run without any crashes. 

Expected results:
Core file was generated.

Additional info:
shd is offline on the node where gluster was upgraded.

Status of volume: distdisp
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick dhcp35-163.lab.eng.blr.redhat.com:/br
icks/brick4/distdisp                        49155     0          Y       5720 
Brick dhcp35-19.lab.eng.blr.redhat.com:/bri
cks/brick4/distdisp                         49154     0          Y       12285
Brick dhcp35-46.lab.eng.blr.redhat.com:/bri
cks/brick4/distdisp                         49154     0          Y       12474
Brick dhcp35-21.lab.eng.blr.redhat.com:/bri
cks/brick4/distdisp                         49155     0          Y       11920
Brick dhcp35-117.lab.eng.blr.redhat.com:/br
icks/brick4/distdisp                        49154     0          Y       12258
Brick dhcp35-86.lab.eng.blr.redhat.com:/bri
cks/brick4/distdisp                         49154     0          Y       17777
Brick dhcp35-163.lab.eng.blr.redhat.com:/br
icks/brick5/distdisp                        49156     0          Y       5729 
Brick dhcp35-19.lab.eng.blr.redhat.com:/bri
cks/brick5/distdisp                         49155     0          Y       12306
Brick dhcp35-46.lab.eng.blr.redhat.com:/bri
cks/brick5/distdisp                         49155     0          Y       12495
Brick dhcp35-21.lab.eng.blr.redhat.com:/bri
cks/brick5/distdisp                         49156     0          Y       11941
Brick dhcp35-117.lab.eng.blr.redhat.com:/br
icks/brick5/distdisp                        49155     0          Y       12279
Brick dhcp35-86.lab.eng.blr.redhat.com:/bri
cks/brick5/distdisp                         49155     0          Y       17798
Self-heal Daemon on localhost               N/A       N/A        N       N/A  
Self-heal Daemon on dhcp35-86.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       18381
Self-heal Daemon on dhcp35-117.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       12807
Self-heal Daemon on dhcp35-21.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       12512
Self-heal Daemon on dhcp35-19.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       12867
Self-heal Daemon on dhcp35-46.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       13082
 
Task Status of Volume distdisp
------------------------------------------------------------------------------
There are no active volume tasks
Comment 4 Atin Mukherjee 2019-04-28 07:31:07 UTC 
Can we please start to investigate it?
Comment 7 Kshithij Iyer 2019-04-28 15:08:12 UTC 
Keeping Atin's needinfo intact.
Comment 22 Atin Mukherjee 2019-06-03 14:00:06 UTC 
Status please?