Bug 1608352
| Summary: | glusterfsd process crashed in a multiplexed configuration during cleanup of a single brick-graph triggered by volume-stop. | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> | |
| Component: | core | Assignee: | Mohit Agrawal <moagrawa> | |
| Status: | CLOSED ERRATA | QA Contact: | Nag Pavan Chilakam <nchilaka> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.4 | CC: | amukherj, apaladug, kdhananj, moagrawa, rhs-bugs, sankarshan, sheggodu, storage-qa-internal, vdas | |
| Target Milestone: | --- | |||
| Target Release: | RHGS 3.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.12.2-16 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1614124 (view as bug list) | Environment: | ||
| Last Closed: | 2018-09-04 06:50:24 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1614124 | |||
| Bug Blocks: | 1503137 | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |
Description of problem: ==================== was running a non-functional sceanrio (steps mentioned in reproducer), which led to brick crash Version-Release number of selected component (if applicable): ------------ 3.12.2-14 How reproducible: ----------------- hit once Steps to Reproduce: 1. created a 6 node cluster 2. created and started 54 1x(2+1) arbiter volumes 3. from n1,terminal1 : trigger of index heal in loop as below #while true;do for i in $(gluster v list);do gluster v heal $i;done;done 4. from same node n1, another terminal; triggered action to stop all volumes in seq, followed by volume start in seq, for i in {1..100};do for v in $(gluster v list);do gluster v stop $v --mode=script;sleep 3;done; for l in $(gluster v list);do gluster v start $l --mode=script;done Actual results(below is bt) ------------------ Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ce/18228a9f759b0b0508ab4f0eaad2bb4421d41a [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s rhs-client30.lab.eng.blr.redhat.com --volfile-id 1-cool'. Program terminated with signal 11, Segmentation fault. #0 0x00007ff402211cba in pl_trace_flush (this=this@entry=0x7ff3dff51260, frame=frame@entry=0x7ff19c001440, fd=fd@entry=0x7ff30c18c090) at common.c:336 336 if (!priv->trace) (gdb) bt #0 0x00007ff402211cba in pl_trace_flush (this=this@entry=0x7ff3dff51260, frame=frame@entry=0x7ff19c001440, fd=fd@entry=0x7ff30c18c090) at common.c:336 #1 0x00007ff40222e435 in pl_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff51260, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at posix.c:1627 #2 0x00007ff415919071 in default_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff52160, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at defaults.c:2263 #3 0x00007ff415919071 in default_flush (frame=0x7ff19c001440, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263 #4 0x00007ff401beffd3 in leases_flush (frame=frame@entry=0x7ff19c000fe0, this=this@entry=0x7ff3dff54400, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at leases.c:920 #5 0x00007ff415919071 in default_flush (frame=0x7ff19c000fe0, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263 #6 0x00007ff415933205 in default_flush_resume (frame=0x7ff3342db870, this=0x7ff3dff564a0, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:1620 #7 0x00007ff4158bf585 in call_resume (stub=0x7ff33414c250) at call-stub.c:2512 #8 0x00007ff4017bfd84 in iot_worker (data=0x7ff3dff816f0) at io-threads.c:224 #9 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3006f9700) at pthread_create.c:308 #10 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb) t a a bt Thread 101 (Thread 0x7ff0675f0700 (LWP 25321)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:205 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0675f0700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 100 (Thread 0x7ff3285f6700 (LWP 25211)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3285f6700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 99 (Thread 0x7ff061ce6700 (LWP 25432)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff3d3418bf8) at changelog-ev-handle.c:205 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff061ce6700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 98 (Thread 0x7ff0a9efc700 (LWP 24998)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0a9efc700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 97 (Thread 0x7ff3482f5700 (LWP 24999)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3482f5700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 96 (Thread 0x7ff3286f7700 (LWP 25208)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3286f7700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 95 (Thread 0x7ff05f4e1700 (LWP 25446)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff3d33c46c0) at posix-helpers.c:1897 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff05f4e1700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 94 (Thread 0x7ff06c6f9700 (LWP 25213)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06c6f9700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 93 (Thread 0x7ff065ded700 (LWP 25324)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff065ded700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 92 (Thread 0x7ff06f7fe700 (LWP 25115)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff3d2e05ae0) at posix-helpers.c:1419 #2 posix_janitor_thread_proc (data=0x7ff3d2e05ae0) at posix-helpers.c:1467 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06f7fe700) at pthread_create.c:308 ---Type <return> to continue, or q <return> to quit--- #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 91 (Thread 0x7ff31c5f5700 (LWP 28243)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dc5a0380) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c5f5700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 90 (Thread 0x7ff067df1700 (LWP 25319)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff067df1700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 89 (Thread 0x7ff0be7fc700 (LWP 24900)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dc5bd858) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0be7fc700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 88 (Thread 0x7ff05fce2700 (LWP 25445)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff3d33c46c0) at posix-helpers.c:2083 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff05fce2700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 87 (Thread 0x7ff0a8efa700 (LWP 25001)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0a8efa700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 86 (Thread 0x7ff07aefe700 (LWP 25103)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:205 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff07aefe700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 85 (Thread 0x7ff07a6fd700 (LWP 25104)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff07a6fd700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 84 (Thread 0x7ff08affd700 (LWP 25013)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dc9dc880) at posix-helpers.c:2083 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff08affd700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 83 (Thread 0x7ff3283f4700 (LWP 25320)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3283f4700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 82 (Thread 0x7ff069ef4700 (LWP 25227)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dd0e6b50) at posix-helpers.c:1419 ---Type <return> to continue, or q <return> to quit--- #2 posix_janitor_thread_proc (data=0x7ff0dd0e6b50) at posix-helpers.c:1467 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff069ef4700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 81 (Thread 0x7ff0604e3700 (LWP 25435)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d3418bf8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0604e3700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 80 (Thread 0x7ff064deb700 (LWP 25332)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dd55a050) at posix-helpers.c:1897 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff064deb700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 79 (Thread 0x7ff0624e7700 (LWP 25430)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0624e7700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 78 (Thread 0x7ff06cefa700 (LWP 25212)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:205 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06cefa700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 77 (Thread 0x7ff06aef6700 (LWP 25225)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dd0e6b50) at posix-helpers.c:2083 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06aef6700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 76 (Thread 0x7ff3287f8700 (LWP 25102)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3287f8700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 75 (Thread 0x7ff05d4dd700 (LWP 28128)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd40490) at syncop.c:603 #2 0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd40490) at syncop.c:695 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff05d4dd700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 74 (Thread 0x7ff348071700 (LWP 25429)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff3d33fb720) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff348071700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 73 (Thread 0x7ff0665ee700 (LWP 25323)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0665ee700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---Type <return> to continue, or q <return> to quit--- Thread 72 (Thread 0x7ff3281f2700 (LWP 25431)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3281f2700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 71 (Thread 0x7ff3484f7700 (LWP 24897)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265bd2b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:375 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3484f7700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 70 (Thread 0x7ff06ffff700 (LWP 25114)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff3d2e05ae0) at posix-helpers.c:1897 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06ffff700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 69 (Thread 0x7ff06d6fb700 (LWP 25210)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06d6fb700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 68 (Thread 0x7ff31c0a0700 (LWP 28289)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dcb5f010) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c0a0700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 67 (Thread 0x7ff349ffb700 (LWP 23586)): #0 0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dcfe0300) at event-epoll.c:649 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff349ffb700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 66 (Thread 0x7ff078efa700 (LWP 25113)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff3d2e05ae0) at posix-helpers.c:2083 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff078efa700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 65 (Thread 0x7ff06effd700 (LWP 25116)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff3d2e05ae0, head=head@entry=0x7ff06effce80) at posix-helpers.c:2151 #2 0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff3d2e05ae0) at posix-helpers.c:2247 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06effd700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 64 (Thread 0x7ff31c122700 (LWP 28280)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd11dc20) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c122700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 63 (Thread 0x7ff060ce4700 (LWP 25434)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d3418bf8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff060ce4700) at pthread_create.c:308 ---Type <return> to continue, or q <return> to quit--- #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 62 (Thread 0x7ff06bef8700 (LWP 25214)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06bef8700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 61 (Thread 0x7ff066def700 (LWP 25322)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd6f9c78) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff066def700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 60 (Thread 0x7ff31c0e1700 (LWP 28285)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff3d33fb720) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c0e1700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 59 (Thread 0x7ff0897fa700 (LWP 25016)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dc9dc880, head=head@entry=0x7ff0897f9e80) at posix-helpers.c:2151 #2 0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dc9dc880) at posix-helpers.c:2247 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0897fa700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 58 (Thread 0x7ff05e4df700 (LWP 25448)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff3d33c46c0, head=head@entry=0x7ff05e4dee80) at posix-helpers.c:2151 #2 0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff3d33c46c0) at posix-helpers.c:2247 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff05e4df700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 57 (Thread 0x7ff06a6f5700 (LWP 25226)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dd0e6b50) at posix-helpers.c:1897 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06a6f5700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 56 (Thread 0x7ff0ab7fe700 (LWP 24911)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dc569300, head=head@entry=0x7ff0ab7fde80) at posix-helpers.c:2151 #2 0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dc569300) at posix-helpers.c:2247 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0ab7fe700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 55 (Thread 0x7ff40c699700 (LWP 23423)): #0 0x00007ff414701411 in do_sigwait (sig=0x7ff40c698e1c, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61 #1 __sigwait (set=set@entry=0x7ff40c698e20, sig=sig@entry=0x7ff40c698e1c) at ../sysdeps/unix/sysv/linux/sigwait.c:99 #2 0x000055585abf352b in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2137 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff40c699700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 54 (Thread 0x7ff3282f3700 (LWP 25428)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3282f3700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---Type <return> to continue, or q <return> to quit--- Thread 53 (Thread 0x7ff07b6ff700 (LWP 25101)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff07b6ff700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 52 (Thread 0x7ff31c42e700 (LWP 28251)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff3d2e3cb00) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c42e700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 51 (Thread 0x7ff39a7fc700 (LWP 23506)): #0 0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3d0476c40) at event-epoll.c:649 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff39a7fc700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 50 (Thread 0x7ff0a96fb700 (LWP 25000)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:205 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0a96fb700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 49 (Thread 0x7ff3600af700 (LWP 24997)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dcb5f010) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3600af700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 48 (Thread 0x7ff3480b2700 (LWP 25318)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd6dc7a0) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3480b2700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 47 (Thread 0x7ff079efc700 (LWP 25105)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff079efc700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 46 (Thread 0x7ff31c573700 (LWP 28245)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd6dc7a0) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c573700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 45 (Thread 0x7ff0614e5700 (LWP 25433)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d3418bf8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0614e5700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 44 (Thread 0x7ff0655ec700 (LWP 25331)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dd55a050) at posix-helpers.c:2083 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0655ec700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---Type <return> to continue, or q <return> to quit--- Thread 43 (Thread 0x7ff4089d1700 (LWP 23427)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff4017bfa0b in iot_exit_threads (conf=conf@entry=0x7ff3dff816f0) at io-threads.c:1067 #2 0x00007ff4017c2010 in fini (this=0x7ff3dff564a0) at io-threads.c:1094 #3 0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff573b0) at glusterfsd-mgmt.c:227 #4 0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff58410) at glusterfsd-mgmt.c:227 #5 0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5a030) at glusterfsd-mgmt.c:227 #6 0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5b280) at glusterfsd-mgmt.c:227 #7 0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5c7e0) at glusterfsd-mgmt.c:227 #8 0x000055585abf62e5 in xlator_call_fini (this=0x7ff3dff5db20) at glusterfsd-mgmt.c:227 #9 0x000055585abf6345 in xlator_mem_cleanup (this=0x7ff3dff5f100) at glusterfsd-mgmt.c:249 #10 0x00007ff4006c51c0 in server_rpc_notify (rpc=<optimized out>, xl=<optimized out>, event=<optimized out>, data=0x7ff3ac152510) at server.c:687 #11 0x00007ff415659125 in rpcsvc_program_notify (listener=0x7ff404041050, event=event@entry=RPCSVC_EVENT_TRANSPORT_DESTROY, data=data@entry=0x7ff3ac152510) at rpcsvc.c:335 #12 0x00007ff41565cb23 in rpcsvc_notify (trans=0x7ff3ac152510, mydata=<optimized out>, event=<optimized out>, data=<optimized out>) at rpcsvc.c:812 #13 0x00007ff41565e9cc in rpc_transport_unref (this=this@entry=0x7ff3ac152510) at rpc-transport.c:517 #14 0x00007ff40a486be7 in socket_event_handler (fd=17, idx=<optimized out>, gen=<optimized out>, data=0x7ff3ac152510, poll_in=<optimized out>, poll_out=0, poll_err=16) at socket.c:2479 #15 0x00007ff4158f84c4 in event_dispatch_epoll_handler (event=0x7ff4089d0e80, event_pool=0x55585cd37210) at event-epoll.c:583 #16 event_dispatch_epoll_worker (data=0x55585cd80e20) at event-epoll.c:659 #17 0x00007ff4146f9dd5 in start_thread (arg=0x7ff4089d1700) at pthread_create.c:308 #18 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 42 (Thread 0x7ff08bfff700 (LWP 25002)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff08bfff700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 41 (Thread 0x7ff05ece0700 (LWP 25447)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff3d33c46c0) at posix-helpers.c:1419 #2 posix_janitor_thread_proc (data=0x7ff3d33c46c0) at posix-helpers.c:1467 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff05ece0700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 40 (Thread 0x7ff31c5b4700 (LWP 28244)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd11dc20) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c5b4700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 39 (Thread 0x7ff31c46f700 (LWP 28250)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dcb5f010) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c46f700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 38 (Thread 0x7ff0bffff700 (LWP 24896)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff40265d303 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:867 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bffff700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 37 (Thread 0x7ff40be98700 (LWP 23424)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff4158c320d in pool_sweeper (arg=<optimized out>) at mem-pool.c:481 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff40be98700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 36 (Thread 0x7ff0696f3700 (LWP 25228)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 ---Type <return> to continue, or q <return> to quit--- #1 0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dd0e6b50, head=head@entry=0x7ff0696f2e80) at posix-helpers.c:2151 #2 0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dd0e6b50) at posix-helpers.c:2247 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0696f3700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 35 (Thread 0x7ff31c267700 (LWP 28263)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff3d33fb720) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31c267700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 34 (Thread 0x7ff3481f4700 (LWP 25099)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3481f4700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 33 (Thread 0x7ff089ffb700 (LWP 25015)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dc9dc880) at posix-helpers.c:1419 #2 posix_janitor_thread_proc (data=0x7ff0dc9dc880) at posix-helpers.c:1467 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff089ffb700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 32 (Thread 0x7ff36006e700 (LWP 25100)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff3d2e3cb00) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff36006e700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 31 (Thread 0x7ff08a7fc700 (LWP 25014)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dc9dc880) at posix-helpers.c:1897 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff08a7fc700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 30 (Thread 0x7ff0bf7fe700 (LWP 24898)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff402a9f893 in changelog_ev_connector (data=0x7ff0dc5bd858) at changelog-ev-handle.c:205 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bf7fe700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 29 (Thread 0x7ff0645ea700 (LWP 25333)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dd55a050) at posix-helpers.c:1419 #2 posix_janitor_thread_proc (data=0x7ff0dd55a050) at posix-helpers.c:1467 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0645ea700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 28 (Thread 0x7ff3284f5700 (LWP 25317)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3284f5700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 27 (Thread 0x7ff0796fb700 (LWP 25106)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff3d2e59fd8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0796fb700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---Type <return> to continue, or q <return> to quit--- Thread 26 (Thread 0x7ff3483f6700 (LWP 24996)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3483f6700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 25 (Thread 0x7ff31f7fe700 (LWP 23630)): #0 0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dd5c0fa0) at event-epoll.c:649 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff31f7fe700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 24 (Thread 0x7ff063de9700 (LWP 25334)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff403ded21b in posix_fsyncer_pick (this=this@entry=0x7ff0dd55a050, head=head@entry=0x7ff063de8e80) at posix-helpers.c:2151 #2 0x00007ff403ded4a5 in posix_fsyncer (d=0x7ff0dd55a050) at posix-helpers.c:2247 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff063de9700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 23 (Thread 0x7ff06b6f7700 (LWP 25215)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dd13b0f8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff06b6f7700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 22 (Thread 0x7ff3480f3700 (LWP 25209)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd11dc20) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3480f3700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 21 (Thread 0x7ff415d7e780 (LWP 23421)): #0 0x00007ff4146faf47 in pthread_join (threadid=140686093260544, thread_return=thread_return@entry=0x0) at pthread_join.c:92 #1 0x00007ff4158f8af8 in event_dispatch_epoll (event_pool=0x55585cd37210) at event-epoll.c:746 #2 0x000055585abf0247 in main (argc=19, argv=<optimized out>) at glusterfsd.c:2550 Thread 20 (Thread 0x7ff0bcff9700 (LWP 24909)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403dec81d in posix_health_check_thread_proc (data=0x7ff0dc569300) at posix-helpers.c:1897 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bcff9700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 19 (Thread 0x7ff3017fa700 (LWP 23678)): #0 0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3d0a3a090) at event-epoll.c:649 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3017fa700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 18 (Thread 0x7ff08b7fe700 (LWP 25003)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dcb7c4e8) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff08b7fe700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 17 (Thread 0x7ff0abfff700 (LWP 24910)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff403de8c25 in janitor_get_next_fd (this=0x7ff0dc569300) at posix-helpers.c:1419 #2 posix_janitor_thread_proc (data=0x7ff0dc569300) at posix-helpers.c:1467 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0abfff700) at pthread_create.c:308 ---Type <return> to continue, or q <return> to quit--- #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 16 (Thread 0x7ff300573700 (LWP 28464)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dc5a0380) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff300573700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7ff3c4ff9700 (LWP 23471)): #0 0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dc476c40) at event-epoll.c:649 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3c4ff9700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 14 (Thread 0x7ff40b697700 (LWP 23425)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd3fd10) at syncop.c:603 #2 0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd3fd10) at syncop.c:695 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff40b697700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 13 (Thread 0x7ff3005f5700 (LWP 28433)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dd6dc7a0) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3005f5700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 12 (Thread 0x7ff3600f0700 (LWP 24895)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff0dc5a0380) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3600f0700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 11 (Thread 0x7ff40ce9a700 (LWP 23422)): #0 0x00007ff414700eed in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158a89d6 in gf_timer_proc (data=0x55585cd3f4f0) at timer.c:174 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff40ce9a700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 10 (Thread 0x7ff0bd7fa700 (LWP 24908)): #0 0x00007ff413f894fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff413f89394 in __sleep (seconds=0, seconds@entry=5) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007ff403ded011 in posix_disk_space_check_thread_proc (data=0x7ff0dc569300) at posix-helpers.c:2083 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bd7fa700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 9 (Thread 0x7ff40ae96700 (LWP 23426)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd400d0) at syncop.c:603 #2 0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd400d0) at syncop.c:695 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff40ae96700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 8 (Thread 0x7ff3485f8700 (LWP 24894)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3485f8700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 7 (Thread 0x7ff360232700 (LWP 24806)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 ---Type <return> to continue, or q <return> to quit--- #1 0x00007ff400f7c065 in index_worker (data=<optimized out>) at index.c:217 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff360232700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 6 (Thread 0x7ff0bdffb700 (LWP 24901)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dc5bd858) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0bdffb700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 5 (Thread 0x7ff0beffd700 (LWP 24899)): #0 0x00007ff413fb9c03 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff402a9fada in changelog_ev_dispatch (data=0x7ff0dc5bd858) at changelog-ev-handle.c:350 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff0beffd700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7ff363fff700 (LWP 23544)): #0 0x00007ff413fc3113 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff4158f8392 in event_dispatch_epoll_worker (data=0x7ff3dca3a090) at event-epoll.c:649 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff363fff700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7ff05ccdc700 (LWP 28323)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4158d5e68 in syncenv_task (proc=proc@entry=0x55585cd40850) at syncop.c:603 #2 0x00007ff4158d6d30 in syncenv_processor (thdata=0x55585cd40850) at syncop.c:695 #3 0x00007ff4146f9dd5 in start_thread (arg=0x7ff05ccdc700) at pthread_create.c:308 #4 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7ff300532700 (LWP 28522)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ff4017bfd3c in iot_worker (data=0x7ff3d2e3cb00) at io-threads.c:193 #2 0x00007ff4146f9dd5 in start_thread (arg=0x7ff300532700) at pthread_create.c:308 #3 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7ff3006f9700 (LWP 28298)): #0 0x00007ff402211cba in pl_trace_flush (this=this@entry=0x7ff3dff51260, frame=frame@entry=0x7ff19c001440, fd=fd@entry=0x7ff30c18c090) at common.c:336 #1 0x00007ff40222e435 in pl_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff51260, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at posix.c:1627 #2 0x00007ff415919071 in default_flush (frame=frame@entry=0x7ff19c001440, this=this@entry=0x7ff3dff52160, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at defaults.c:2263 #3 0x00007ff415919071 in default_flush (frame=0x7ff19c001440, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263 #4 0x00007ff401beffd3 in leases_flush (frame=frame@entry=0x7ff19c000fe0, this=this@entry=0x7ff3dff54400, fd=fd@entry=0x7ff30c18c090, xdata=xdata@entry=0x0) at leases.c:920 #5 0x00007ff415919071 in default_flush (frame=0x7ff19c000fe0, this=<optimized out>, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:2263 #6 0x00007ff415933205 in default_flush_resume (frame=0x7ff3342db870, this=0x7ff3dff564a0, fd=0x7ff30c18c090, xdata=0x0) at defaults.c:1620 #7 0x00007ff4158bf585 in call_resume (stub=0x7ff33414c250) at call-stub.c:2512 #8 0x00007ff4017bfd84 in iot_worker (data=0x7ff3dff816f0) at io-threads.c:224 #9 0x00007ff4146f9dd5 in start_thread (arg=0x7ff3006f9700) at pthread_create.c:308 #10 0x00007ff413fc2b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113