Bug 1460919
| Summary: | Dead lock hit due to wrong order of locking in open behind and hence leading to fuse mount and client node hung | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> |
| Component: | open-behind | Assignee: | Susant Kumar Palai <spalai> |
| Status: | CLOSED DEFERRED | QA Contact: | Rahul Hinduja <rhinduja> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | rhgs-3.3 | CC: | amukherj, mchangir, rgowdapp, rhs-bugs, storage-qa-internal |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | --- | Flags: | ykaul:
needinfo-
ykaul: needinfo- ykaul: needinfo- |
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-12-31 07:21:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
RPM and other test info at http://etherpad.corp.redhat.com/nchilaka-Discoverer-Accelerated-Testing https://docs.google.com/spreadsheets/d/1ik5nqPJB1wtoFjhkFpk2RPcGBuL76yQ-gaQ-WPR_atI/edit#gid=0 Two mount process were deadlocked and one of them had active inodelks on a brick. Both the deadlocks involved interaction between open-behind and itable dump code Deadlock1: statedump had acquired locks in the order: itable -> inode -> fd openbehind was trying to acquire locks in the order: fd -> itable (as part of inode_ref done after acquiring fd->lock) Deadlock2: statedump was trying to acquire lock: itable -> inode openbehind was trying to acquire lock: inode -> itable (it was trying to create an anonymous fd which involves doing an inode_ref after acquiring inode->lock) Both the deadlocks are present in upstream master. Is this bug addressed in upstream now? Or should we track this in upstream? Closing - it received no real activity for ages. Please re-open if relevant and will be taken care of. |
Description of problem: ======================== I was running different IOs from multiple clients as part of a hotfix below is what was being done from the clients which hit the deadlock c1:was taking statedumps and moving them to the mount point every 5 min, along with this I was doing a gzip of old log files [root@dhcp35-65 logs]# tar -cvf dhcp37-141.lab.eng.blr.redhat.com.tar dhcp37-141.lab.eng.blr.redhat.com 2>>err.$HOSTNAME.tarOfdhcp37-141.lab.eng.blr.redhat.com.log C2:was taking statedumps and moving them to mount point every 5 min , along with that i was taring old log files rhs-client14: dhcp35-205.lab.eng.blr.redhat.com/statedumps/glusterdump.1126.dump.1496441621 dhcp35-205.lab.eng.blr.redhat.com/statedumps/glusterdump.1126.dump.1496442228 Information from brick statedumps: [root@dhcp35-38 gluster]# grep ACTIVE *dump* rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=d039d0f12b7f0000, client=0x7fccc8424740, connection-id=dhcp37-104.lab.eng.blr.redhat.com-19802-2017/06/07-08:33:32:166149-distrep-client-0-0-0, granted at 2017-06-12 11:16:06 rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=387c1fa0b47f0000, client=0x7fccc81e1f70, connection-id=dhcp35-65.lab.eng.blr.redhat.com-2247-2017/06/09-11:21:21:750837-distrep-client-0-0-0, granted at 2017-06-12 11:11:56 rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=201f89d0c27f0000, client=0x7fccc032f3d0, connection-id=dhcp35-38.lab.eng.blr.redhat.com-1536-2017/06/07-08:33:29:741310-distrep-client-0-0-0, granted at 2017-06-09 14:28:58 rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=987a1fd82b7f0000, client=0x7fccc85f7c70, connection-id=rhs-client14.lab.eng.blr.redhat.com-28312-2017/06/09-12:01:58:954220-distrep-client-0-0-0, granted at 2017-06-09 14:24:12 rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=d039d0f12b7f0000, client=0x7fccc8424740, connection-id=dhcp37-104.lab.eng.blr.redhat.com-19802-2017/06/07-08:33:32:166149-distrep-client-0-0-0, granted at 2017-06-12 11:16:06 rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=387c1fa0b47f0000, client=0x7fccc81e1f70, connection-id=dhcp35-65.lab.eng.blr.redhat.com-2247-2017/06/09-11:21:21:750837-distrep-client-0-0-0, granted at 2017-06-12 11:11:56 rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=201f89d0c27f0000, client=0x7fccc032f3d0, connection-id=dhcp35-38.lab.eng.blr.redhat.com-1536-2017/06/07-08:33:29:741310-distrep-client-0-0-0, granted at 2017-06-09 14:28:58 rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=987a1fd82b7f0000, client=0x7fccc85f7c70, connection-id=rhs-client14.lab.eng.blr.redhat.com-28312-2017/06/09-12:01:58:954220-distrep-client-0-0-0, granted at 2017-06-09 14:24:12 Statedump which hung on client 10.70.35.65: (gdb) bt #0 0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762 #2 0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370 (gdb) thr apply all bt Thread 9 (Thread 0x7fc161055700 (LWP 1127)): #0 0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc160654700 (LWP 1128)): #0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 #1 0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 #3 0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389 #4 0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028 #5 0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503 #6 0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010) at statedump.c:829 #7 0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699 #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 #4 0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695 #5 0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122 #6 0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0) at open-behind.c:159 #7 0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999 #8 0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at io-cache.c:610 #9 0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at read-ahead.c:106 #10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0) at dht-inode-read.c:64 ---Type <return> to continue, or q <return> to quit--- #11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, fd=<value optimized out>, xdata=0x0) at afr-open.c:110 #12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469 #13 0x00007fc169aec6c5 in rpc_clnt_handle_reply (clnt=0x7fc1345b5c70, pollin=0x7fc103fc2c80) at rpc-clnt.c:765 #14 0x00007fc169aed7f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7fc1345b5ca0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fc103fc2c80) at rpc-clnt.c:938 #15 0x00007fc169ae8b38 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546 #16 0x00007fc15e6437ed in socket_event_poll_in (this=0x7fc12d1d5bd0) at socket.c:2353 #17 0x00007fc15e644abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7fc12d1d5bd0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466 #18 0x00007fc169d92080 in event_dispatch_epoll_handler (data=0x7fc16b80d2a0) at event-epoll.c:575 #19 event_dispatch_epoll_worker (data=0x7fc16b80d2a0) at event-epoll.c:678 #20 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #21 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc155d59700 (LWP 1132)): #0 0x00007fc168777163 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc169d91eb1 in event_dispatch_epoll_worker (data=0x7fc158079fe0) at event-epoll.c:668 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 4 (Thread 0x7fc14ffff700 (LWP 1135)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4d171 in inode_ref (inode=0x7fc101152d4c) at inode.c:545 #4 0x00007fc16105ca6f in fuse_ino_to_inode (ino=140466923580748, fuse=<value optimized out>) at fuse-helpers.c:390 #5 0x00007fc16105da1d in fuse_resolve_inode_init (state=<value optimized out>, resolve=0x7fc12a375800, ino=<value optimized out>) at fuse-resolve.c:621 #6 0x00007fc161067f6a in fuse_getattr (this=0x7fc16b7d3fc0, finh=0x7fc14a2e8d10, msg=<value optimized out>) at fuse-bridge.c:854 #7 0x00007fc161074c70 in fuse_thread_proc (data=0x7fc16b7d3fc0) at fuse-bridge.c:4986 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc14f5fe700 (LWP 1136)): #0 0x00007fc168e0c5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc1610679db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc14ebfd700 (LWP 2784)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7a90) at syncop.c:607 #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7a90) at syncop.c:699 ---Type <return> to continue, or q <return> to quit--- #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc16a1e0740 (LWP 1126)): #0 0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762 #2 0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370 (gdb) thr apply all bt Thread 9 (Thread 0x7fc161055700 (LWP 1127)): #0 0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc160654700 (LWP 1128)): #0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 #1 0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 #3 0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389 #4 0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028 #5 0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503 #6 0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010) at statedump.c:829 #7 0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699 #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 #4 0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695 #5 0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122 #6 0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0) at open-behind.c:159 #7 0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999 #8 0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at io-cache.c:610 #9 0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at read-ahead.c:106 #10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0) at dht-inode-read.c:64 ---Type <return> to continue, or q <return> to quit--- #11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, fd=<value optimized out>, xdata=0x0) at afr-open.c:110 #12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469 #13 0x00007fc169aec6c5 in rpc_clnt_handle_reply (clnt=0x7fc1345b5c70, pollin=0x7fc103fc2c80) at rpc-clnt.c:765 #14 0x00007fc169aed7f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7fc1345b5ca0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fc103fc2c80) at rpc-clnt.c:938 #15 0x00007fc169ae8b38 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546 #16 0x00007fc15e6437ed in socket_event_poll_in (this=0x7fc12d1d5bd0) at socket.c:2353 #17 0x00007fc15e644abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7fc12d1d5bd0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466 #18 0x00007fc169d92080 in event_dispatch_epoll_handler (data=0x7fc16b80d2a0) at event-epoll.c:575 #19 event_dispatch_epoll_worker (data=0x7fc16b80d2a0) at event-epoll.c:678 #20 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #21 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc155d59700 (LWP 1132)): #0 0x00007fc168777163 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc169d91eb1 in event_dispatch_epoll_worker (data=0x7fc158079fe0) at event-epoll.c:668 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 4 (Thread 0x7fc14ffff700 (LWP 1135)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4d171 in inode_ref (inode=0x7fc101152d4c) at inode.c:545 #4 0x00007fc16105ca6f in fuse_ino_to_inode (ino=140466923580748, fuse=<value optimized out>) at fuse-helpers.c:390 #5 0x00007fc16105da1d in fuse_resolve_inode_init (state=<value optimized out>, resolve=0x7fc12a375800, ino=<value optimized out>) at fuse-resolve.c:621 #6 0x00007fc161067f6a in fuse_getattr (this=0x7fc16b7d3fc0, finh=0x7fc14a2e8d10, msg=<value optimized out>) at fuse-bridge.c:854 #7 0x00007fc161074c70 in fuse_thread_proc (data=0x7fc16b7d3fc0) at fuse-bridge.c:4986 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc14f5fe700 (LWP 1136)): #0 0x00007fc168e0c5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc1610679db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc14ebfd700 (LWP 2784)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7a90) at syncop.c:607 #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7a90) at syncop.c:699 ---Type <return> to continue, or q <return> to quit--- #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc16a1e0740 (LWP 1126)): #0 0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762 #2 0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370 (gdb) thr apply all bt Thread 9 (Thread 0x7fc161055700 (LWP 1127)): #0 0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc160654700 (LWP 1128)): #0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 #1 0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 #3 0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389 #4 0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028 #5 0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503 #6 0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010) at statedump.c:829 #7 0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) thr 8 [Switching to thread 8 (Thread 0x7fc160654700 (LWP 1128))]#0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 (gdb) f 2 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 2337 fd_ctx_dump (fd, prefix); (gdb) l 2332 if (dump_options.xl_options.dump_fdctx != _gf_true) 2333 goto unlock; 2334 2335 2336 list_for_each_entry (fd, &inode->fd_list, inode_list) { 2337 fd_ctx_dump (fd, prefix); 2338 } 2339 } 2340 unlock: 2341 UNLOCK(&inode->lock); (gdb) p inode->gfid $1 = "D\025cI\231\364I4\266\235)}\356!E", <incomplete sequence \314> (gdb) p inode $2 = (inode_t *) 0x7fc101125650 (gdb) thr apply all bt Thread 9 (Thread 0x7fc161055700 (LWP 1127)): #0 0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc160654700 (LWP 1128)): #0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 #1 0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 #3 0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389 #4 0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028 #5 0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503 #6 0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010) at statedump.c:829 #7 0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699 #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 #4 0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695 #5 0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122 #6 0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0) at open-behind.c:159 #7 0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999 #8 0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at io-cache.c:610 #9 0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at read-ahead.c:106 #10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0) at dht-inode-read.c:64 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) thr 6 [Switching to thread 6 (Thread 0x7fc15d01f700 (LWP 1131))]#0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) f 3 #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 523 pthread_mutex_lock (&table->lock); (gdb) thr 8 [Switching to thread 8 (Thread 0x7fc160654700 (LWP 1128))]#0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 (gdb) f 2 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 2337 fd_ctx_dump (fd, prefix); (gdb) thr 6 [Switching to thread 6 (Thread 0x7fc15d01f700 (LWP 1131))]#0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 #4 0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695 #5 0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122 #6 0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0) at open-behind.c:159 #7 0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999 #8 0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at io-cache.c:610 #9 0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at read-ahead.c:106 #10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0) at dht-inode-read.c:64 #11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, fd=<value optimized out>, xdata=0x0) at afr-open.c:110 #12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) f 6 #6 0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0) at open-behind.c:159 159 ob_fd_free (ob_fd); (gdb) l 154 if (op_ret < 0) { 155 /* mark fd BAD for ever */ 156 ob_fd->op_errno = op_errno; 157 } else { 158 __fd_ctx_del (fd, this, NULL); 159 ob_fd_free (ob_fd); 160 } 161 } 162 UNLOCK (&fd->lock); 163 (gdb) down #5 0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122 122 loc_wipe (&ob_fd->loc); (gdb) down #4 0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695 695 inode_unref (loc->inode); (gdb) down #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 523 pthread_mutex_lock (&table->lock); (gdb) down #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 (gdb) down #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 (gdb) thr apply all bt Thread 9 (Thread 0x7fc161055700 (LWP 1127)): #0 0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc160654700 (LWP 1128)): #0 0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0 #1 0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137 #2 0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337 #3 0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389 #4 0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028 #5 0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503 #6 0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010) at statedump.c:829 #7 0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699 #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523 #4 0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695 #5 0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122 #6 0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0) at open-behind.c:159 #7 0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999 #8 0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at io-cache.c:610 #9 0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at read-ahead.c:106 #10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0) at dht-inode-read.c:64 ---Type <return> to continue, or q <return> to quit--- #11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, fd=<value optimized out>, xdata=0x0) at afr-open.c:110 #12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469 #13 0x00007fc169aec6c5 in rpc_clnt_handle_reply (clnt=0x7fc1345b5c70, pollin=0x7fc103fc2c80) at rpc-clnt.c:765 #14 0x00007fc169aed7f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7fc1345b5ca0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fc103fc2c80) at rpc-clnt.c:938 #15 0x00007fc169ae8b38 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546 #16 0x00007fc15e6437ed in socket_event_poll_in (this=0x7fc12d1d5bd0) at socket.c:2353 #17 0x00007fc15e644abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7fc12d1d5bd0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466 #18 0x00007fc169d92080 in event_dispatch_epoll_handler (data=0x7fc16b80d2a0) at event-epoll.c:575 #19 event_dispatch_epoll_worker (data=0x7fc16b80d2a0) at event-epoll.c:678 #20 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #21 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc155d59700 (LWP 1132)): #0 0x00007fc168777163 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc169d91eb1 in event_dispatch_epoll_worker (data=0x7fc158079fe0) at event-epoll.c:668 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 4 (Thread 0x7fc14ffff700 (LWP 1135)): #0 0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc169d4d171 in inode_ref (inode=0x7fc101152d4c) at inode.c:545 #4 0x00007fc16105ca6f in fuse_ino_to_inode (ino=140466923580748, fuse=<value optimized out>) at fuse-helpers.c:390 #5 0x00007fc16105da1d in fuse_resolve_inode_init (state=<value optimized out>, resolve=0x7fc12a375800, ino=<value optimized out>) at fuse-resolve.c:621 #6 0x00007fc161067f6a in fuse_getattr (this=0x7fc16b7d3fc0, finh=0x7fc14a2e8d10, msg=<value optimized out>) at fuse-bridge.c:854 #7 0x00007fc161074c70 in fuse_thread_proc (data=0x7fc16b7d3fc0) at fuse-bridge.c:4986 #8 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc14f5fe700 (LWP 1136)): #0 0x00007fc168e0c5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc1610679db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873 #2 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc14ebfd700 (LWP 2784)): #0 0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7a90) at syncop.c:607 #2 0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7a90) at syncop.c:699 ---Type <return> to continue, or q <return> to quit--- #3 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc168776b6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc16a1e0740 (LWP 1126)): #0 0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762 #2 0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370 (gdb) ################################################ rhs-client14:gdb of hung fuse process [root@rhs-client14 ~]# gdb -p 28314 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/cluster/replicate.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/cluster/afr.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/cluster/replicate.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/cluster/distribute.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/cluster/dht.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/cluster/distribute.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/write-behind.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/write-behind.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/write-behind.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/read-ahead.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/read-ahead.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/read-ahead.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/readdir-ahead.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/readdir-ahead.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/readdir-ahead.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/io-cache.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/io-cache.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/io-cache.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/quick-read.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/quick-read.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/quick-read.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/open-behind.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/open-behind.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/open-behind.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/debug/io-stats.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/debug/io-stats.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/debug/io-stats.so Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/meta.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/meta.so.debug...done. done. Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/meta.so Reading symbols from /lib64/libgcc_s.so.1... warning: the debug information found in "/usr/lib/debug//lib64/libgcc_s-4.4.7-20120601.so.1.debug" does not match "/lib64/libgcc_s.so.1" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug" does not match "/lib64/libgcc_s.so.1" (CRC mismatch). (no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89 89 lll_wait_tid (pd->tid); Missing separate debuginfos, use: debuginfo-install keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-57.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-18.el6.x86_64 libselinux-2.0.94-7.el6.x86_64 libuuid-2.17.2-12.24.el6.x86_64 openssl-1.0.1e-48.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) (gdb) (gdb) bt #0 0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89 #1 0x00007f2beb357b2d in event_dispatch_epoll (event_pool=0x7f2bed9bcd30) at event-epoll.c:762 #2 0x00007f2beb7c5181 in main (argc=4, argv=0x7fffccaa5438) at glusterfsd.c:2370 (gdb) #0 0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89 #1 0x00007f2beb357b2d in event_dispatch_epoll (event_pool=0x7f2bed9bcd30) at event-epoll.c:762 #2 0x00007f2beb7c5181 in main (argc=4, argv=0x7fffccaa5438) at glusterfsd.c:2370 (gdb) t a a bt Thread 9 (Thread 0x7f2be2614700 (LWP 28315)): #0 0x00007f2bea3d600d in nanosleep () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2beb30fcb2 in gf_timer_proc (ctx=0x7f2bed99e010) at timer.c:193 #2 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be2614700) at pthread_create.c:301 #3 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 8 (Thread 0x7f2be1c13700 (LWP 28316)): #0 0x00007f2bea3d3462 in ?? () from /lib64/libpthread.so.0 #1 0x00007f2beb3110ba in inode_ctx_get0 (inode=0x7f2bd546e3e8, xlator=<value optimized out>, value1=<value optimized out>) at inode.c:2126 #2 0x00007f2bdcc6cc27 in ioc_inode_dump (this=0x7f2bd80181a0, inode=0x7f2bd546e3e8) at io-cache.c:1991 #3 0x00007f2beb310428 in inode_dump (inode=0x7f2bd546e3e8, prefix=<value optimized out>) at inode.c:2348 #4 0x00007f2beb31212f in inode_table_dump (itable=0x7f2bd80022f0, prefix=0x7f2be26382fd "xlator.mount.fuse.itable") at inode.c:2389 #5 0x00007f2be2628e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028 #6 0x00007f2beb32d0e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503 #7 0x00007f2beb32d9d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7f2bed99e010) at statedump.c:829 #8 0x00007f2beb7c321d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035 #9 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be1c13700) at pthread_create.c:301 #10 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 7 (Thread 0x7f2be1212700 (LWP 28317)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239 #1 0x00007f2beb33836b in syncenv_task (proc=0x7f2bed9decf0) at syncop.c:607 #2 0x00007f2beb33da10 in syncenv_processor (thdata=0x7f2bed9decf0) at syncop.c:699 #3 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be1212700) at pthread_create.c:301 #4 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 6 (Thread 0x7f2be0811700 (LWP 28318)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239 #1 0x00007f2beb33836b in syncenv_task (proc=0x7f2bed9df0b0) at syncop.c:607 #2 0x00007f2beb33da10 in syncenv_processor (thdata=0x7f2bed9df0b0) at syncop.c:699 #3 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be0811700) at pthread_create.c:301 #4 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 5 (Thread 0x7f2bde5dc700 (LWP 28320)): #0 0x00007f2be9d380a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2beb357eb1 in event_dispatch_epoll_worker (data=0x7f2beda05170) at event-epoll.c:668 #2 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bde5dc700) at pthread_create.c:301 #3 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 4 (Thread 0x7f2bd654c700 (LWP 28321)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136 #1 0x00007f2bea3d05d8 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007f2bea3d04a7 in __pthread_mutex_lock (mutex=0x7f2bd80022f0) at pthread_mutex_lock.c:61 #3 0x00007f2beb3104e1 in inode_is_linked (inode=0x7f2bd5428650) at inode.c:2283 #4 0x00007f2bdd743b37 in __afr_inode_write_finalize (frame=0x7f2be8ea14d4, this=0x7f2bd800f310) at afr-inode-write.c:59 #5 0x00007f2bdd745ee2 in afr_writev_wind_cbk (frame=0x7f2be8ea14d4, cookie=<value optimized out>, this=0x7f2bd800f310, op_ret=61440, op_errno=<value optimized out>, prebuf=<value optimized out>, postbuf=0x7f2bd654ba50, xdata=0x7f2be88c098c) at afr-inode-write.c:352 #6 0x00007f2bdd9c457f in client3_3_writev_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7f2be8e9f948) at client-rpc-fops.c:912 #7 0x00007f2beb0b26c5 in rpc_clnt_handle_reply (clnt=0x7f2bd81e54d0, pollin=0x7f2bd0016da0) at rpc-clnt.c:765 #8 0x00007f2beb0b37f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7f2bd81e5500, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f2bd0016da0) at rpc-clnt.c:938 #9 0x00007f2beb0aeb38 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546 #10 0x00007f2bdfc027ed in socket_event_poll_in (this=0x7f2bd81f51c0) at socket.c:2353 #11 0x00007f2bdfc03abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7f2bd81f51c0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466 #12 0x00007f2beb358080 in event_dispatch_epoll_handler (data=0x7f2bd80c0dd0) at event-epoll.c:575 ---Type <return> to continue, or q <return> to quit--- #13 event_dispatch_epoll_worker (data=0x7f2bd80c0dd0) at event-epoll.c:678 #14 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bd654c700) at pthread_create.c:301 #15 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 3 (Thread 0x7f2bd5041700 (LWP 28324)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136 #1 0x00007f2bea3d05d8 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007f2bea3d04a7 in __pthread_mutex_lock (mutex=0x7f2bd80022f0) at pthread_mutex_lock.c:61 #3 0x00007f2beb313171 in inode_ref (inode=0x7f2bd546e3e8) at inode.c:545 #4 0x00007f2beb3251e7 in __fd_create (inode=0x7f2bd546e3e8, pid=0) at fd.c:636 #5 0x00007f2beb3252c5 in __fd_anonymous (inode=0x7f2bd546e3e8, flags=2) at fd.c:782 #6 0x00007f2beb3253aa in fd_anonymous (inode=0x7f2bd546e3e8) at fd.c:806 #7 0x00007f2bdc857118 in ob_readv (frame=0x7f2be8ea8054, this=0x7f2bd801a8a0, fd=<value optimized out>, size=131072, offset=<value optimized out>, flags=<value optimized out>, xdata=0x0) at open-behind.c:381 #8 0x00007f2bdc642213 in mdc_readv (frame=0x7f2be8ea37c4, this=0x7f2bd801bc50, fd=0x7f2bd81f7b70, size=131072, offset=31571968, flags=32768, xdata=0x0) at md-cache.c:1550 #9 0x00007f2bdc426706 in io_stats_readv (frame=0x7f2be8ea799c, this=0x7f2bd801d060, fd=0x7f2bd81f7b70, size=131072, offset=31571968, flags=32768, xdata=0x0) at io-stats.c:2141 #10 0x00007f2beb2eed2f in default_readv (frame=0x7f2be8ea799c, this=0x7f2bd801e5a0, fd=0x7f2bd81f7b70, size=131072, offset=31571968, flags=32768, xdata=0x0) at defaults.c:1846 #11 0x00007f2bdc20f4ec in meta_readv (frame=0x7f2be8ea799c, this=0x7f2bd801e5a0, fd=0x7f2bd81f7b70, size=131072, offset=<value optimized out>, flags=<value optimized out>, xdata=0x0) at meta.c:79 #12 0x00007f2be2624d30 in fuse_readv_resume (state=<value optimized out>) at fuse-bridge.c:2242 #13 0x00007f2be261db65 in fuse_resolve_done (state=<value optimized out>) at fuse-resolve.c:668 #14 fuse_resolve_all (state=<value optimized out>) at fuse-resolve.c:695 #15 0x00007f2be261d896 in fuse_resolve (state=0x7f2bc801cbe0) at fuse-resolve.c:659 #16 0x00007f2be261dbae in fuse_resolve_all (state=<value optimized out>) at fuse-resolve.c:691 #17 0x00007f2be261dc13 in fuse_resolve_continue (state=<value optimized out>) at fuse-resolve.c:711 #18 0x00007f2be261d83e in fuse_resolve_fd (state=0x7f2bc801cbe0) at fuse-resolve.c:571 #19 fuse_resolve (state=0x7f2bc801cbe0) at fuse-resolve.c:648 #20 0x00007f2be261db8e in fuse_resolve_all (state=<value optimized out>) at fuse-resolve.c:684 #21 0x00007f2be261dbd8 in fuse_resolve_and_resume (state=0x7f2bc801cbe0, fn=0x7f2be2624aa0 <fuse_readv_resume>) at fuse-resolve.c:723 #22 0x00007f2be2633c70 in fuse_thread_proc (data=0x7f2bed9cb760) at fuse-bridge.c:4986 #23 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bd5041700) at pthread_create.c:301 #24 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 2 (Thread 0x7f2bcffff700 (LWP 28325)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183 #1 0x00007f2be26269db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873 #2 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bcffff700) at pthread_create.c:301 #3 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 1 (Thread 0x7f2beb7a5740 (LWP 28314)): #0 0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89 #1 0x00007f2beb357b2d in event_dispatch_epoll (event_pool=0x7f2bed9bcd30) at event-epoll.c:762 #2 0x00007f2beb7c5181 in main (argc=4, argv=0x7fffccaa5438) at glusterfsd.c:2370 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: