Description of problem:
====================
was running a non-functional sceanrio (steps mentioned in reproducer), which led to brick crash




Version-Release number of selected component (if applicable):
------------
3.12.2-14

How reproducible:
-----------------
hit once

Steps to Reproduce:
1. created a 6 node cluster
2. created and started 54 1x(2+1) arbiter volumes
3. from n1,terminal1 : trigger of index heal in loop as below
 #while true;do for i in $(gluster v list);do gluster v heal $i;done;done

4. from same node n1, another terminal; triggered action to stop all volumes in seq, followed by volume start in seq, 
for  i  in {1..100};do for v in $(gluster v list);do gluster v stop $v --mode=script;sleep 3;done; for l in $(gluster v list);do gluster v start $l --mode=script;done





Actual results(below is bt)
------------------

Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ce/18228a9f759b0b0508ab4f0eaad2bb4421d41a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s rhs-client30.lab.eng.blr.redhat.com --volfile-id 1-cool'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007ff402211cba in pl_trace_flush (this=this@entry=0x7ff3dff51260, frame=frame@entry=0x7ff19c001440, fd=fd@entry=0x7ff30c18c090) at common.c:336
336	        if (!priv->trace)
(gdb) bt
#0  0x00007ff402211cba in pl_trace_flush (this=this@entry=0x7ff3dff51260, frame=frame@entry=0x7ff19c001440, fd=fd@entry=0x7ff30c18c090) at common.c:336
#1  0x00007ff40222e435 in pl_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff51260, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at posix.c:1627
#2  0x00007ff415919071 in default_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff52160, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at defaults.c:2263
#3  0x00007ff415919071 in default_flush (frame=0x7ff19c001440, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263
#4  0x00007ff401beffd3 in leases_flush (frame=frame@entry=0x7ff19c000fe0, this=this@entry=0x7ff3dff54400, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at leases.c:920
#5  0x00007ff415919071 in default_flush (frame=0x7ff19c000fe0, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263
#6  0x00007ff415933205 in default_flush_resume (frame=0x7ff3342db870, this=0x7ff3dff564a0, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:1620
#7  0x00007ff4158bf585 in call_resume (stub=0x7ff33414c250) at call-stub.c:2512
#8  0x00007ff4017bfd84 in iot_worker (data=0x7ff3dff816f0) at io-threads.c:224
#9  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3006f9700) at pthread_create.c:308
#10 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) t a a bt

Thread 101 (Thread 0x7ff0675f0700 (LWP 25321)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:205
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0675f0700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 100 (Thread 0x7ff3285f6700 (LWP 25211)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3285f6700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 99 (Thread 0x7ff061ce6700 (LWP 25432)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff3d3418bf8) at changelog-ev-handle.c:205
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff061ce6700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 98 (Thread 0x7ff0a9efc700 (LWP 24998)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0a9efc700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 97 (Thread 0x7ff3482f5700 (LWP 24999)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3482f5700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 96 (Thread 0x7ff3286f7700 (LWP 25208)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3286f7700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 95 (Thread 0x7ff05f4e1700 (LWP 25446)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff3d33c46c0) at posix-helpers.c:1897
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff05f4e1700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 94 (Thread 0x7ff06c6f9700 (LWP 25213)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06c6f9700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 93 (Thread 0x7ff065ded700 (LWP 25324)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff065ded700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 92 (Thread 0x7ff06f7fe700 (LWP 25115)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff3d2e05ae0) at posix-helpers.c:1419
#2  posix_janitor_thread_proc (data=0x7ff3d2e05ae0) at posix-helpers.c:1467
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06f7fe700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 91 (Thread 0x7ff31c5f5700 (LWP 28243)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dc5a0380) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c5f5700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 90 (Thread 0x7ff067df1700 (LWP 25319)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff067df1700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 89 (Thread 0x7ff0be7fc700 (LWP 24900)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dc5bd858) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0be7fc700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 88 (Thread 0x7ff05fce2700 (LWP 25445)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff3d33c46c0) at posix-helpers.c:2083
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff05fce2700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 87 (Thread 0x7ff0a8efa700 (LWP 25001)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0a8efa700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 86 (Thread 0x7ff07aefe700 (LWP 25103)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:205
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff07aefe700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 85 (Thread 0x7ff07a6fd700 (LWP 25104)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff07a6fd700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 84 (Thread 0x7ff08affd700 (LWP 25013)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dc9dc880) at posix-helpers.c:2083
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff08affd700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 83 (Thread 0x7ff3283f4700 (LWP 25320)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3283f4700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 82 (Thread 0x7ff069ef4700 (LWP 25227)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dd0e6b50) at posix-helpers.c:1419
---Type <return> to continue, or q <return> to quit---
#2  posix_janitor_thread_proc (data=0x7ff0dd0e6b50) at posix-helpers.c:1467
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff069ef4700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 81 (Thread 0x7ff0604e3700 (LWP 25435)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d3418bf8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0604e3700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 80 (Thread 0x7ff064deb700 (LWP 25332)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dd55a050) at posix-helpers.c:1897
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff064deb700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 79 (Thread 0x7ff0624e7700 (LWP 25430)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0624e7700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 78 (Thread 0x7ff06cefa700 (LWP 25212)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:205
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06cefa700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 77 (Thread 0x7ff06aef6700 (LWP 25225)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dd0e6b50) at posix-helpers.c:2083
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06aef6700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 76 (Thread 0x7ff3287f8700 (LWP 25102)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3287f8700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 75 (Thread 0x7ff05d4dd700 (LWP 28128)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd40490) at syncop.c:603
#2  0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd40490) at syncop.c:695
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff05d4dd700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 74 (Thread 0x7ff348071700 (LWP 25429)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff3d33fb720) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff348071700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 73 (Thread 0x7ff0665ee700 (LWP 25323)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0665ee700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

---Type <return> to continue, or q <return> to quit---
Thread 72 (Thread 0x7ff3281f2700 (LWP 25431)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3281f2700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 71 (Thread 0x7ff3484f7700 (LWP 24897)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3484f7700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 70 (Thread 0x7ff06ffff700 (LWP 25114)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff3d2e05ae0) at posix-helpers.c:1897
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06ffff700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 69 (Thread 0x7ff06d6fb700 (LWP 25210)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06d6fb700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 68 (Thread 0x7ff31c0a0700 (LWP 28289)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dcb5f010) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c0a0700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 67 (Thread 0x7ff349ffb700 (LWP 23586)):
#0  0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dcfe0300) at event-epoll.c:649
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff349ffb700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 66 (Thread 0x7ff078efa700 (LWP 25113)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff3d2e05ae0) at posix-helpers.c:2083
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff078efa700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 65 (Thread 0x7ff06effd700 (LWP 25116)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff3d2e05ae0, head=head@entry=0x7ff06effce80) at posix-helpers.c:2151
#2  0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff3d2e05ae0) at posix-helpers.c:2247
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06effd700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 64 (Thread 0x7ff31c122700 (LWP 28280)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd11dc20) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c122700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 63 (Thread 0x7ff060ce4700 (LWP 25434)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d3418bf8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff060ce4700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 62 (Thread 0x7ff06bef8700 (LWP 25214)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06bef8700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 61 (Thread 0x7ff066def700 (LWP 25322)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff066def700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 60 (Thread 0x7ff31c0e1700 (LWP 28285)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff3d33fb720) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c0e1700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 59 (Thread 0x7ff0897fa700 (LWP 25016)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dc9dc880, head=head@entry=0x7ff0897f9e80) at posix-helpers.c:2151
#2  0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dc9dc880) at posix-helpers.c:2247
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0897fa700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 58 (Thread 0x7ff05e4df700 (LWP 25448)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff3d33c46c0, head=head@entry=0x7ff05e4dee80) at posix-helpers.c:2151
#2  0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff3d33c46c0) at posix-helpers.c:2247
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff05e4df700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 57 (Thread 0x7ff06a6f5700 (LWP 25226)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dd0e6b50) at posix-helpers.c:1897
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06a6f5700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 56 (Thread 0x7ff0ab7fe700 (LWP 24911)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dc569300, head=head@entry=0x7ff0ab7fde80) at posix-helpers.c:2151
#2  0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dc569300) at posix-helpers.c:2247
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0ab7fe700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 55 (Thread 0x7ff40c699700 (LWP 23423)):
#0  0x00007ff414701411 in do_sigwait (sig=0x7ff40c698e1c, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1  __sigwait (set=set@entry=0x7ff40c698e20, sig=sig@entry=0x7ff40c698e1c) at ../sysdeps/unix/sysv/linux/sigwait.c:99
#2  0x000055585abf352b in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2137
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff40c699700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 54 (Thread 0x7ff3282f3700 (LWP 25428)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3282f3700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

---Type <return> to continue, or q <return> to quit---
Thread 53 (Thread 0x7ff07b6ff700 (LWP 25101)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff07b6ff700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 52 (Thread 0x7ff31c42e700 (LWP 28251)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff3d2e3cb00) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c42e700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 51 (Thread 0x7ff39a7fc700 (LWP 23506)):
#0  0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3d0476c40) at event-epoll.c:649
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff39a7fc700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 50 (Thread 0x7ff0a96fb700 (LWP 25000)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:205
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0a96fb700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 49 (Thread 0x7ff3600af700 (LWP 24997)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dcb5f010) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3600af700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 48 (Thread 0x7ff3480b2700 (LWP 25318)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd6dc7a0) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3480b2700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 47 (Thread 0x7ff079efc700 (LWP 25105)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff079efc700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 46 (Thread 0x7ff31c573700 (LWP 28245)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd6dc7a0) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c573700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 45 (Thread 0x7ff0614e5700 (LWP 25433)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d3418bf8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0614e5700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 44 (Thread 0x7ff0655ec700 (LWP 25331)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dd55a050) at posix-helpers.c:2083
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0655ec700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

---Type <return> to continue, or q <return> to quit---
Thread 43 (Thread 0x7ff4089d1700 (LWP 23427)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff4017bfa0b in iot_exit_threads (conf=conf@entry=0x7ff3dff816f0) at io-threads.c:1067
#2  0x00007ff4017c2010 in fini (this=0x7ff3dff564a0) at io-threads.c:1094
#3  0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff573b0) at glusterfsd-mgmt.c:227
#4  0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff58410) at glusterfsd-mgmt.c:227
#5  0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5a030) at glusterfsd-mgmt.c:227
#6  0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5b280) at glusterfsd-mgmt.c:227
#7  0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5c7e0) at glusterfsd-mgmt.c:227
#8  0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5db20) at glusterfsd-mgmt.c:227
#9  0x000055585abf6345 in xlator_mem_cleanup (this=0x7ff3dff5f100) at glusterfsd-mgmt.c:249
#10 0x00007ff4006c51c0 in server_rpc_notify (rpc=<optimized out>, xl=<optimized out>, event=<optimized out>, data=0x7ff3ac152510) at server.c:687
#11 0x00007ff415659125 in rpcsvc_program_notify (listener=0x7ff404041050, event=event@entry=RPCSVC_EVENT_TRANSPORT_DESTROY, data=data@entry=0x7ff3ac152510) at rpcsvc.c:335
#12 0x00007ff41565cb23 in rpcsvc_notify (trans=0x7ff3ac152510, mydata=<optimized out>, event=<optimized out>, data=<optimized out>) at rpcsvc.c:812
#13 0x00007ff41565e9cc in rpc_transport_unref (this=this@entry=0x7ff3ac152510) at rpc-transport.c:517
#14 0x00007ff40a486be7 in socket_event_handler (fd=17, idx=<optimized out>, gen=<optimized out>, data=0x7ff3ac152510, poll_in=<optimized out>, poll_out=0, poll_err=16) at socket.c:2479
#15 0x00007ff4158f84c4 in event_dispatch_epoll_handler (event=0x7ff4089d0e80, event_pool=0x55585cd37210) at event-epoll.c:583
#16 event_dispatch_epoll_worker (data=0x55585cd80e20) at event-epoll.c:659
#17 0x00007ff4146f9dd5 in start_thread (arg=0x7ff4089d1700) at pthread_create.c:308
#18 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 42 (Thread 0x7ff08bfff700 (LWP 25002)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff08bfff700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 41 (Thread 0x7ff05ece0700 (LWP 25447)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff3d33c46c0) at posix-helpers.c:1419
#2  posix_janitor_thread_proc (data=0x7ff3d33c46c0) at posix-helpers.c:1467
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff05ece0700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 40 (Thread 0x7ff31c5b4700 (LWP 28244)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd11dc20) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c5b4700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 39 (Thread 0x7ff31c46f700 (LWP 28250)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dcb5f010) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c46f700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 38 (Thread 0x7ff0bffff700 (LWP 24896)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bffff700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 37 (Thread 0x7ff40be98700 (LWP 23424)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff4158c320d in pool_sweeper (arg=<optimized out>) at mem-pool.c:481
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff40be98700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 36 (Thread 0x7ff0696f3700 (LWP 25228)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
---Type <return> to continue, or q <return> to quit---
#1  0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dd0e6b50, head=head@entry=0x7ff0696f2e80) at posix-helpers.c:2151
#2  0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dd0e6b50) at posix-helpers.c:2247
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0696f3700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 35 (Thread 0x7ff31c267700 (LWP 28263)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff3d33fb720) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c267700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 34 (Thread 0x7ff3481f4700 (LWP 25099)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3481f4700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 33 (Thread 0x7ff089ffb700 (LWP 25015)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dc9dc880) at posix-helpers.c:1419
#2  posix_janitor_thread_proc (data=0x7ff0dc9dc880) at posix-helpers.c:1467
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff089ffb700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 32 (Thread 0x7ff36006e700 (LWP 25100)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff3d2e3cb00) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff36006e700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 31 (Thread 0x7ff08a7fc700 (LWP 25014)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dc9dc880) at posix-helpers.c:1897
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff08a7fc700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 30 (Thread 0x7ff0bf7fe700 (LWP 24898)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dc5bd858) at changelog-ev-handle.c:205
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bf7fe700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 29 (Thread 0x7ff0645ea700 (LWP 25333)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dd55a050) at posix-helpers.c:1419
#2  posix_janitor_thread_proc (data=0x7ff0dd55a050) at posix-helpers.c:1467
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0645ea700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 28 (Thread 0x7ff3284f5700 (LWP 25317)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3284f5700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 27 (Thread 0x7ff0796fb700 (LWP 25106)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0796fb700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
---Type <return> to continue, or q <return> to quit---

Thread 26 (Thread 0x7ff3483f6700 (LWP 24996)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3483f6700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 25 (Thread 0x7ff31f7fe700 (LWP 23630)):
#0  0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dd5c0fa0) at event-epoll.c:649
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff31f7fe700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 24 (Thread 0x7ff063de9700 (LWP 25334)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dd55a050, head=head@entry=0x7ff063de8e80) at posix-helpers.c:2151
#2  0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dd55a050) at posix-helpers.c:2247
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff063de9700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 23 (Thread 0x7ff06b6f7700 (LWP 25215)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff06b6f7700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 22 (Thread 0x7ff3480f3700 (LWP 25209)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd11dc20) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3480f3700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 21 (Thread 0x7ff415d7e780 (LWP 23421)):
#0  0x00007ff4146faf47 in pthread_join (threadid=140686093260544, thread_return=thread_return@entry=0x0) at pthread_join.c:92
#1  0x00007ff4158f8af8 in event_dispatch_epoll (event_pool=0x55585cd37210) at event-epoll.c:746
#2  0x000055585abf0247 in main (argc=19, argv=<optimized out>) at glusterfsd.c:2550

Thread 20 (Thread 0x7ff0bcff9700 (LWP 24909)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dc569300) at posix-helpers.c:1897
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bcff9700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 19 (Thread 0x7ff3017fa700 (LWP 23678)):
#0  0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3d0a3a090) at event-epoll.c:649
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3017fa700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 18 (Thread 0x7ff08b7fe700 (LWP 25003)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff08b7fe700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 17 (Thread 0x7ff0abfff700 (LWP 24910)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dc569300) at posix-helpers.c:1419
#2  posix_janitor_thread_proc (data=0x7ff0dc569300) at posix-helpers.c:1467
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0abfff700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 16 (Thread 0x7ff300573700 (LWP 28464)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dc5a0380) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff300573700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 15 (Thread 0x7ff3c4ff9700 (LWP 23471)):
#0  0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dc476c40) at event-epoll.c:649
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3c4ff9700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 14 (Thread 0x7ff40b697700 (LWP 23425)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd3fd10) at syncop.c:603
#2  0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd3fd10) at syncop.c:695
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff40b697700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 13 (Thread 0x7ff3005f5700 (LWP 28433)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd6dc7a0) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3005f5700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 12 (Thread 0x7ff3600f0700 (LWP 24895)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff0dc5a0380) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3600f0700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 11 (Thread 0x7ff40ce9a700 (LWP 23422)):
#0  0x00007ff414700eed in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158a89d6 in gf_timer_proc (data=0x55585cd3f4f0) at timer.c:174
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff40ce9a700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 10 (Thread 0x7ff0bd7fa700 (LWP 24908)):
#0  0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dc569300) at posix-helpers.c:2083
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bd7fa700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 9 (Thread 0x7ff40ae96700 (LWP 23426)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd400d0) at syncop.c:603
#2  0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd400d0) at syncop.c:695
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff40ae96700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 8 (Thread 0x7ff3485f8700 (LWP 24894)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3485f8700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 7 (Thread 0x7ff360232700 (LWP 24806)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
---Type <return> to continue, or q <return> to quit---
#1  0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff360232700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 6 (Thread 0x7ff0bdffb700 (LWP 24901)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dc5bd858) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bdffb700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 5 (Thread 0x7ff0beffd700 (LWP 24899)):
#0  0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dc5bd858) at changelog-ev-handle.c:350
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff0beffd700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7ff363fff700 (LWP 23544)):
#0  0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dca3a090) at event-epoll.c:649
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff363fff700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7ff05ccdc700 (LWP 28323)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd40850) at syncop.c:603
#2  0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd40850) at syncop.c:695
#3  0x00007ff4146f9dd5 in start_thread (arg=0x7ff05ccdc700) at pthread_create.c:308
#4  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 2 (Thread 0x7ff300532700 (LWP 28522)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff4017bfd3c in iot_worker (data=0x7ff3d2e3cb00) at io-threads.c:193
#2  0x00007ff4146f9dd5 in start_thread (arg=0x7ff300532700) at pthread_create.c:308
#3  0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x7ff3006f9700 (LWP 28298)):
#0  0x00007ff402211cba in pl_trace_flush (this=this@entry=0x7ff3dff51260, frame=frame@entry=0x7ff19c001440, fd=fd@entry=0x7ff30c18c090) at common.c:336
#1  0x00007ff40222e435 in pl_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff51260, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at posix.c:1627
#2  0x00007ff415919071 in default_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff52160, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at defaults.c:2263
#3  0x00007ff415919071 in default_flush (frame=0x7ff19c001440, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263
#4  0x00007ff401beffd3 in leases_flush (frame=frame@entry=0x7ff19c000fe0, this=this@entry=0x7ff3dff54400, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at leases.c:920
#5  0x00007ff415919071 in default_flush (frame=0x7ff19c000fe0, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263
#6  0x00007ff415933205 in default_flush_resume (frame=0x7ff3342db870, this=0x7ff3dff564a0, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:1620
#7  0x00007ff4158bf585 in call_resume (stub=0x7ff33414c250) at call-stub.c:2512
#8  0x00007ff4017bfd84 in iot_worker (data=0x7ff3dff816f0) at io-threads.c:224
#9  0x00007ff4146f9dd5 in start_thread (arg=0x7ff3006f9700) at pthread_create.c:308
#10 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113