Bug 1460919 - Dead lock hit due to wrong order of locking in open behind and hence leading to fuse mount and client node hung [NEEDINFO]
Dead lock hit due to wrong order of locking in open behind and hence leading ...
Status: ASSIGNED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: open-behind (Show other bugs)
3.3
Unspecified Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Milind Changire
Rahul Hinduja
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-13 03:17 EDT by nchilaka
Modified: 2018-03-10 13:19 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
mchangir: needinfo? (rgowdapp)


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2017-06-13 03:17:08 EDT
Description of problem:
========================
I was running different IOs from multiple clients as part of a hotfix
below is what was being done from the clients which hit the deadlock
c1:was taking statedumps and moving them to the mount point every 5 min, along with this I was doing a gzip of old log files

[root@dhcp35-65 logs]# tar -cvf dhcp37-141.lab.eng.blr.redhat.com.tar dhcp37-141.lab.eng.blr.redhat.com 2>>err.$HOSTNAME.tarOfdhcp37-141.lab.eng.blr.redhat.com.log


C2:was taking statedumps and moving them to mount point every 5 min , along with that i was taring old log files
rhs-client14:
dhcp35-205.lab.eng.blr.redhat.com/statedumps/glusterdump.1126.dump.1496441621
dhcp35-205.lab.eng.blr.redhat.com/statedumps/glusterdump.1126.dump.1496442228



Information from brick statedumps:
[root@dhcp35-38 gluster]# grep ACTIVE *dump*
rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=d039d0f12b7f0000, client=0x7fccc8424740, connection-id=dhcp37-104.lab.eng.blr.redhat.com-19802-2017/06/07-08:33:32:166149-distrep-client-0-0-0, granted at 2017-06-12 11:16:06
rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=387c1fa0b47f0000, client=0x7fccc81e1f70, connection-id=dhcp35-65.lab.eng.blr.redhat.com-2247-2017/06/09-11:21:21:750837-distrep-client-0-0-0, granted at 2017-06-12 11:11:56
rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=201f89d0c27f0000, client=0x7fccc032f3d0, connection-id=dhcp35-38.lab.eng.blr.redhat.com-1536-2017/06/07-08:33:29:741310-distrep-client-0-0-0, granted at 2017-06-09 14:28:58
rhs-brick1-distrep.14279.dump.1497334083:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=987a1fd82b7f0000, client=0x7fccc85f7c70, connection-id=rhs-client14.lab.eng.blr.redhat.com-28312-2017/06/09-12:01:58:954220-distrep-client-0-0-0, granted at 2017-06-09 14:24:12
rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=d039d0f12b7f0000, client=0x7fccc8424740, connection-id=dhcp37-104.lab.eng.blr.redhat.com-19802-2017/06/07-08:33:32:166149-distrep-client-0-0-0, granted at 2017-06-12 11:16:06
rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=387c1fa0b47f0000, client=0x7fccc81e1f70, connection-id=dhcp35-65.lab.eng.blr.redhat.com-2247-2017/06/09-11:21:21:750837-distrep-client-0-0-0, granted at 2017-06-12 11:11:56
rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 18446744073709551610, owner=201f89d0c27f0000, client=0x7fccc032f3d0, connection-id=dhcp35-38.lab.eng.blr.redhat.com-1536-2017/06/07-08:33:29:741310-distrep-client-0-0-0, granted at 2017-06-09 14:28:58
rhs-brick1-distrep.14279.dump.1497335789:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 0, owner=987a1fd82b7f0000, client=0x7fccc85f7c70, connection-id=rhs-client14.lab.eng.blr.redhat.com-28312-2017/06/09-12:01:58:954220-distrep-client-0-0-0, granted at 2017-06-09 14:24:12





Statedump which hung on client 10.70.35.65:
(gdb) bt
#0  0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762
#2  0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370
(gdb) thr apply all bt

Thread 9 (Thread 0x7fc161055700 (LWP 1127)):
#0  0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fc160654700 (LWP 1128)):
#0  0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0
#1  0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
#3  0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, 
    prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389
#4  0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028
#5  0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503
#6  0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010)
    at statedump.c:829
#7  0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607
---Type <return> to continue, or q <return> to quit---
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
#4  0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695
#5  0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122
#6  0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, 
    this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0)
    at open-behind.c:159
#7  0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999
#8  0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at io-cache.c:610
#9  0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at read-ahead.c:106
#10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0)
    at dht-inode-read.c:64
---Type <return> to continue, or q <return> to quit---
#11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, 
    fd=<value optimized out>, xdata=0x0) at afr-open.c:110
#12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, 
    count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469
#13 0x00007fc169aec6c5 in rpc_clnt_handle_reply (clnt=0x7fc1345b5c70, pollin=0x7fc103fc2c80)
    at rpc-clnt.c:765
#14 0x00007fc169aed7f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7fc1345b5ca0, 
    event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fc103fc2c80) at rpc-clnt.c:938
#15 0x00007fc169ae8b38 in rpc_transport_notify (this=<value optimized out>, 
    event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546
#16 0x00007fc15e6437ed in socket_event_poll_in (this=0x7fc12d1d5bd0) at socket.c:2353
#17 0x00007fc15e644abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, 
    data=0x7fc12d1d5bd0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466
#18 0x00007fc169d92080 in event_dispatch_epoll_handler (data=0x7fc16b80d2a0) at event-epoll.c:575
#19 event_dispatch_epoll_worker (data=0x7fc16b80d2a0) at event-epoll.c:678
#20 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fc155d59700 (LWP 1132)):
#0  0x00007fc168777163 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc169d91eb1 in event_dispatch_epoll_worker (data=0x7fc158079fe0) at event-epoll.c:668
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 4 (Thread 0x7fc14ffff700 (LWP 1135)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4d171 in inode_ref (inode=0x7fc101152d4c) at inode.c:545
#4  0x00007fc16105ca6f in fuse_ino_to_inode (ino=140466923580748, fuse=<value optimized out>)
    at fuse-helpers.c:390
#5  0x00007fc16105da1d in fuse_resolve_inode_init (state=<value optimized out>, resolve=0x7fc12a375800, 
    ino=<value optimized out>) at fuse-resolve.c:621
#6  0x00007fc161067f6a in fuse_getattr (this=0x7fc16b7d3fc0, finh=0x7fc14a2e8d10, 
    msg=<value optimized out>) at fuse-bridge.c:854
#7  0x00007fc161074c70 in fuse_thread_proc (data=0x7fc16b7d3fc0) at fuse-bridge.c:4986
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc14f5fe700 (LWP 1136)):
#0  0x00007fc168e0c5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc1610679db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc14ebfd700 (LWP 2784)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7a90) at syncop.c:607
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7a90) at syncop.c:699
---Type <return> to continue, or q <return> to quit---
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc16a1e0740 (LWP 1126)):
#0  0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762
#2  0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370
(gdb) thr apply all bt

Thread 9 (Thread 0x7fc161055700 (LWP 1127)):
#0  0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fc160654700 (LWP 1128)):
#0  0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0
#1  0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
#3  0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, 
    prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389
#4  0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028
#5  0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503
#6  0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010)
    at statedump.c:829
#7  0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607
---Type <return> to continue, or q <return> to quit---
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
#4  0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695
#5  0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122
#6  0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, 
    this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0)
    at open-behind.c:159
#7  0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999
#8  0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at io-cache.c:610
#9  0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at read-ahead.c:106
#10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0)
    at dht-inode-read.c:64
---Type <return> to continue, or q <return> to quit---
#11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, 
    fd=<value optimized out>, xdata=0x0) at afr-open.c:110
#12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, 
    count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469
#13 0x00007fc169aec6c5 in rpc_clnt_handle_reply (clnt=0x7fc1345b5c70, pollin=0x7fc103fc2c80)
    at rpc-clnt.c:765
#14 0x00007fc169aed7f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7fc1345b5ca0, 
    event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fc103fc2c80) at rpc-clnt.c:938
#15 0x00007fc169ae8b38 in rpc_transport_notify (this=<value optimized out>, 
    event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546
#16 0x00007fc15e6437ed in socket_event_poll_in (this=0x7fc12d1d5bd0) at socket.c:2353
#17 0x00007fc15e644abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, 
    data=0x7fc12d1d5bd0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466
#18 0x00007fc169d92080 in event_dispatch_epoll_handler (data=0x7fc16b80d2a0) at event-epoll.c:575
#19 event_dispatch_epoll_worker (data=0x7fc16b80d2a0) at event-epoll.c:678
#20 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fc155d59700 (LWP 1132)):
#0  0x00007fc168777163 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc169d91eb1 in event_dispatch_epoll_worker (data=0x7fc158079fe0) at event-epoll.c:668
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 4 (Thread 0x7fc14ffff700 (LWP 1135)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4d171 in inode_ref (inode=0x7fc101152d4c) at inode.c:545
#4  0x00007fc16105ca6f in fuse_ino_to_inode (ino=140466923580748, fuse=<value optimized out>)
    at fuse-helpers.c:390
#5  0x00007fc16105da1d in fuse_resolve_inode_init (state=<value optimized out>, resolve=0x7fc12a375800, 
    ino=<value optimized out>) at fuse-resolve.c:621
#6  0x00007fc161067f6a in fuse_getattr (this=0x7fc16b7d3fc0, finh=0x7fc14a2e8d10, 
    msg=<value optimized out>) at fuse-bridge.c:854
#7  0x00007fc161074c70 in fuse_thread_proc (data=0x7fc16b7d3fc0) at fuse-bridge.c:4986
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc14f5fe700 (LWP 1136)):
#0  0x00007fc168e0c5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc1610679db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc14ebfd700 (LWP 2784)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7a90) at syncop.c:607
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7a90) at syncop.c:699
---Type <return> to continue, or q <return> to quit---
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc16a1e0740 (LWP 1126)):
#0  0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762
#2  0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370
(gdb) thr apply all bt

Thread 9 (Thread 0x7fc161055700 (LWP 1127)):
#0  0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fc160654700 (LWP 1128)):
#0  0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0
#1  0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
#3  0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, 
    prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389
#4  0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028
#5  0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503
#6  0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010)
    at statedump.c:829
#7  0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) thr 8
[Switching to thread 8 (Thread 0x7fc160654700 (LWP 1128))]#0  0x00007fc168e0d392 in ?? ()
   from /lib64/libpthread.so.0
(gdb) f 2
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
2337	                        fd_ctx_dump (fd, prefix);
(gdb) l
2332			if (dump_options.xl_options.dump_fdctx != _gf_true)
2333				goto unlock;
2334	
2335	
2336	                list_for_each_entry (fd, &inode->fd_list, inode_list) {
2337	                        fd_ctx_dump (fd, prefix);
2338	                }
2339	        }
2340	unlock:
2341	        UNLOCK(&inode->lock);
(gdb) p inode->gfid
$1 = "D\025cI\231\364I4\266\235)}\356!E", <incomplete sequence \314>
(gdb) p inode
$2 = (inode_t *) 0x7fc101125650
(gdb) thr apply all bt

Thread 9 (Thread 0x7fc161055700 (LWP 1127)):
#0  0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fc160654700 (LWP 1128)):
#0  0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0
#1  0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
#3  0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, 
    prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389
#4  0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028
#5  0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503
#6  0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010)
    at statedump.c:829
#7  0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607
---Type <return> to continue, or q <return> to quit---
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
#4  0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695
#5  0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122
#6  0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, 
    this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0)
    at open-behind.c:159
#7  0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999
#8  0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at io-cache.c:610
#9  0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at read-ahead.c:106
#10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0)
    at dht-inode-read.c:64
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) thr 6
[Switching to thread 6 (Thread 0x7fc15d01f700 (LWP 1131))]#0  0x00007fc168e0f264 in __lll_lock_wait ()
   from /lib64/libpthread.so.0
(gdb) f 3
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
523	        pthread_mutex_lock (&table->lock);
(gdb) thr 8
[Switching to thread 8 (Thread 0x7fc160654700 (LWP 1128))]#0  0x00007fc168e0d392 in ?? ()
   from /lib64/libpthread.so.0
(gdb) f 2
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
2337	                        fd_ctx_dump (fd, prefix);
(gdb) thr 6
[Switching to thread 6 (Thread 0x7fc15d01f700 (LWP 1131))]#0  0x00007fc168e0f264 in __lll_lock_wait ()
   from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
#4  0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695
#5  0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122
#6  0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, 
    this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0)
    at open-behind.c:159
#7  0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999
#8  0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at io-cache.c:610
#9  0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at read-ahead.c:106
#10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0)
    at dht-inode-read.c:64
#11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, 
    fd=<value optimized out>, xdata=0x0) at afr-open.c:110
#12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, 
    count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) f 6
#6  0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, 
    this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0)
    at open-behind.c:159
159				ob_fd_free (ob_fd);
(gdb) l
154			if (op_ret < 0) {
155				/* mark fd BAD for ever */
156				ob_fd->op_errno = op_errno;
157			} else {
158				__fd_ctx_del (fd, this, NULL);
159				ob_fd_free (ob_fd);
160			}
161		}
162		UNLOCK (&fd->lock);
163	
(gdb) down
#5  0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122
122		loc_wipe (&ob_fd->loc);
(gdb) down
#4  0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695
695	                inode_unref (loc->inode);
(gdb) down
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
523	        pthread_mutex_lock (&table->lock);
(gdb) down
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
(gdb) down
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
(gdb) thr apply all bt

Thread 9 (Thread 0x7fc161055700 (LWP 1127)):
#0  0x00007fc168e0ff3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fc169d49cb2 in gf_timer_proc (ctx=0x7fc16b7a6010) at timer.c:193
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fc160654700 (LWP 1128)):
#0  0x00007fc168e0d392 in ?? () from /lib64/libpthread.so.0
#1  0x00007fc169d5e96d in fd_ctx_dump (fd=0x7fc109ddb90c, prefix=<value optimized out>) at fd.c:1137
#2  0x00007fc169d4a3db in inode_dump (inode=0x7fc101125650, 
    prefix=0x7fc16064d3f0 "xlator.mount.fuse.itable.active.1") at inode.c:2337
#3  0x00007fc169d4c12f in inode_table_dump (itable=0x7fc12e024430, 
    prefix=0x7fc1610792fd "xlator.mount.fuse.itable") at inode.c:2389
#4  0x00007fc161069e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028
#5  0x00007fc169d670e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503
#6  0x00007fc169d679d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7fc16b7a6010)
    at statedump.c:829
#7  0x00007fc16a1fc21d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fc15fc53700 (LWP 1129)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7310) at syncop.c:607
---Type <return> to continue, or q <return> to quit---
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7310) at syncop.c:699
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc15d01f700 (LWP 1131)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4ce11 in inode_unref (inode=0x7fc101125650) at inode.c:523
#4  0x00007fc169d1c2d2 in loc_wipe (loc=0x7fc12a11b5c8) at xlator.c:695
#5  0x00007fc1571386da in ob_fd_free (ob_fd=0x7fc12a11b5c0) at open-behind.c:122
#6  0x00007fc1571388ba in ob_wake_cbk (frame=0x7fc1678dff74, cookie=<value optimized out>, 
    this=0x7fc137f20580, op_ret=0, op_errno=117, fd_ret=<value optimized out>, xdata=0x0)
    at open-behind.c:159
#7  0x00007fc169d33c15 in default_open_cbk (frame=0x7fc1678dd7d0, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=0x7fc109ddb90c, xdata=0x0) at defaults.c:999
#8  0x00007fc157549a4a in ioc_open_cbk (frame=0x7fc1678e1d04, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at io-cache.c:610
#9  0x00007fc157965460 in ra_open_cbk (frame=0x7fc1678e7304, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=117, fd=0x7fc109ddb90c, 
    xdata=0x0) at read-ahead.c:106
#10 0x00007fc157de28d1 in dht_open_cbk (frame=0x7fc1678e164c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=0, op_errno=117, fd=<value optimized out>, xdata=0x0)
    at dht-inode-read.c:64
---Type <return> to continue, or q <return> to quit---
#11 0x00007fc15c1907d0 in afr_open_cbk (frame=0x7fc1678de290, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, 
    fd=<value optimized out>, xdata=0x0) at afr-open.c:110
#12 0x00007fc15c3fa937 in client3_3_open_cbk (req=<value optimized out>, iov=<value optimized out>, 
    count=<value optimized out>, myframe=0x7fc1678e2c78) at client-rpc-fops.c:469
#13 0x00007fc169aec6c5 in rpc_clnt_handle_reply (clnt=0x7fc1345b5c70, pollin=0x7fc103fc2c80)
    at rpc-clnt.c:765
#14 0x00007fc169aed7f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7fc1345b5ca0, 
    event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fc103fc2c80) at rpc-clnt.c:938
#15 0x00007fc169ae8b38 in rpc_transport_notify (this=<value optimized out>, 
    event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546
#16 0x00007fc15e6437ed in socket_event_poll_in (this=0x7fc12d1d5bd0) at socket.c:2353
#17 0x00007fc15e644abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, 
    data=0x7fc12d1d5bd0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466
#18 0x00007fc169d92080 in event_dispatch_epoll_handler (data=0x7fc16b80d2a0) at event-epoll.c:575
#19 event_dispatch_epoll_worker (data=0x7fc16b80d2a0) at event-epoll.c:678
#20 0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fc155d59700 (LWP 1132)):
#0  0x00007fc168777163 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc169d91eb1 in event_dispatch_epoll_worker (data=0x7fc158079fe0) at event-epoll.c:668
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 4 (Thread 0x7fc14ffff700 (LWP 1135)):
#0  0x00007fc168e0f264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc168e0a508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc168e0a3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc169d4d171 in inode_ref (inode=0x7fc101152d4c) at inode.c:545
#4  0x00007fc16105ca6f in fuse_ino_to_inode (ino=140466923580748, fuse=<value optimized out>)
    at fuse-helpers.c:390
#5  0x00007fc16105da1d in fuse_resolve_inode_init (state=<value optimized out>, resolve=0x7fc12a375800, 
    ino=<value optimized out>) at fuse-resolve.c:621
#6  0x00007fc161067f6a in fuse_getattr (this=0x7fc16b7d3fc0, finh=0x7fc14a2e8d10, 
    msg=<value optimized out>) at fuse-bridge.c:854
#7  0x00007fc161074c70 in fuse_thread_proc (data=0x7fc16b7d3fc0) at fuse-bridge.c:4986
#8  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc14f5fe700 (LWP 1136)):
#0  0x00007fc168e0c5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc1610679db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873
#2  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc14ebfd700 (LWP 2784)):
#0  0x00007fc168e0c98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc169d7236b in syncenv_task (proc=0x7fc16b7e7a90) at syncop.c:607
#2  0x00007fc169d77a10 in syncenv_processor (thdata=0x7fc16b7e7a90) at syncop.c:699
---Type <return> to continue, or q <return> to quit---
#3  0x00007fc168e089d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc168776b6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc16a1e0740 (LWP 1126)):
#0  0x00007fc168e0922d in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fc169d91b2d in event_dispatch_epoll (event_pool=0x7fc16b7c55d0) at event-epoll.c:762
#2  0x00007fc16a1fe181 in main (argc=4, argv=0x7fff0378fe68) at glusterfsd.c:2370
(gdb) 


################################################
rhs-client14:gdb of hung fuse process
[root@rhs-client14 ~]# gdb -p 28314


Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/cluster/replicate.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/cluster/afr.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/cluster/replicate.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/cluster/distribute.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/cluster/dht.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/cluster/distribute.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/write-behind.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/write-behind.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/write-behind.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/read-ahead.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/read-ahead.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/read-ahead.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/readdir-ahead.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/readdir-ahead.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/readdir-ahead.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/io-cache.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/io-cache.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/io-cache.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/quick-read.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/quick-read.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/quick-read.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/open-behind.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/open-behind.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/open-behind.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/debug/io-stats.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/debug/io-stats.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/debug/io-stats.so
Reading symbols from /usr/lib64/glusterfs/3.7.9/xlator/meta.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.7.9/xlator/meta.so.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.7.9/xlator/meta.so
Reading symbols from /lib64/libgcc_s.so.1...
warning: the debug information found in "/usr/lib/debug//lib64/libgcc_s-4.4.7-20120601.so.1.debug" does not match "/lib64/libgcc_s.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug" does not match "/lib64/libgcc_s.so.1" (CRC mismatch).

(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89
89	    lll_wait_tid (pd->tid);
Missing separate debuginfos, use: debuginfo-install keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-57.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-18.el6.x86_64 libselinux-2.0.94-7.el6.x86_64 libuuid-2.17.2-12.24.el6.x86_64 openssl-1.0.1e-48.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) 
(gdb) 
(gdb) bt
#0  0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89
#1  0x00007f2beb357b2d in event_dispatch_epoll (event_pool=0x7f2bed9bcd30) at event-epoll.c:762
#2  0x00007f2beb7c5181 in main (argc=4, argv=0x7fffccaa5438) at glusterfsd.c:2370
(gdb) 
#0  0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89
#1  0x00007f2beb357b2d in event_dispatch_epoll (event_pool=0x7f2bed9bcd30) at event-epoll.c:762
#2  0x00007f2beb7c5181 in main (argc=4, argv=0x7fffccaa5438) at glusterfsd.c:2370
(gdb) t a a bt

Thread 9 (Thread 0x7f2be2614700 (LWP 28315)):
#0  0x00007f2bea3d600d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2beb30fcb2 in gf_timer_proc (ctx=0x7f2bed99e010) at timer.c:193
#2  0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be2614700) at pthread_create.c:301
#3  0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7f2be1c13700 (LWP 28316)):
#0  0x00007f2bea3d3462 in ?? () from /lib64/libpthread.so.0
#1  0x00007f2beb3110ba in inode_ctx_get0 (inode=0x7f2bd546e3e8, xlator=<value optimized out>, value1=<value optimized out>) at inode.c:2126
#2  0x00007f2bdcc6cc27 in ioc_inode_dump (this=0x7f2bd80181a0, inode=0x7f2bd546e3e8) at io-cache.c:1991
#3  0x00007f2beb310428 in inode_dump (inode=0x7f2bd546e3e8, prefix=<value optimized out>) at inode.c:2348
#4  0x00007f2beb31212f in inode_table_dump (itable=0x7f2bd80022f0, prefix=0x7f2be26382fd "xlator.mount.fuse.itable") at inode.c:2389
#5  0x00007f2be2628e69 in fuse_itable_dump (this=<value optimized out>) at fuse-bridge.c:5028
#6  0x00007f2beb32d0e2 in gf_proc_dump_xlator_info (top=<value optimized out>) at statedump.c:503
#7  0x00007f2beb32d9d8 in gf_proc_dump_info (signum=<value optimized out>, ctx=0x7f2bed99e010) at statedump.c:829
#8  0x00007f2beb7c321d in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:2035
#9  0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be1c13700) at pthread_create.c:301
#10 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7f2be1212700 (LWP 28317)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007f2beb33836b in syncenv_task (proc=0x7f2bed9decf0) at syncop.c:607
#2  0x00007f2beb33da10 in syncenv_processor (thdata=0x7f2bed9decf0) at syncop.c:699
#3  0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be1212700) at pthread_create.c:301
#4  0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7f2be0811700 (LWP 28318)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007f2beb33836b in syncenv_task (proc=0x7f2bed9df0b0) at syncop.c:607
#2  0x00007f2beb33da10 in syncenv_processor (thdata=0x7f2bed9df0b0) at syncop.c:699
#3  0x00007f2bea3ceaa1 in start_thread (arg=0x7f2be0811700) at pthread_create.c:301
#4  0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7f2bde5dc700 (LWP 28320)):
#0  0x00007f2be9d380a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2beb357eb1 in event_dispatch_epoll_worker (data=0x7f2beda05170) at event-epoll.c:668
#2  0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bde5dc700) at pthread_create.c:301
#3  0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7f2bd654c700 (LWP 28321)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00007f2bea3d05d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007f2bea3d04a7 in __pthread_mutex_lock (mutex=0x7f2bd80022f0) at pthread_mutex_lock.c:61
#3  0x00007f2beb3104e1 in inode_is_linked (inode=0x7f2bd5428650) at inode.c:2283
#4  0x00007f2bdd743b37 in __afr_inode_write_finalize (frame=0x7f2be8ea14d4, this=0x7f2bd800f310) at afr-inode-write.c:59
#5  0x00007f2bdd745ee2 in afr_writev_wind_cbk (frame=0x7f2be8ea14d4, cookie=<value optimized out>, this=0x7f2bd800f310, op_ret=61440, op_errno=<value optimized out>, prebuf=<value optimized out>, 
    postbuf=0x7f2bd654ba50, xdata=0x7f2be88c098c) at afr-inode-write.c:352
#6  0x00007f2bdd9c457f in client3_3_writev_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7f2be8e9f948) at client-rpc-fops.c:912
#7  0x00007f2beb0b26c5 in rpc_clnt_handle_reply (clnt=0x7f2bd81e54d0, pollin=0x7f2bd0016da0) at rpc-clnt.c:765
#8  0x00007f2beb0b37f1 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7f2bd81e5500, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f2bd0016da0) at rpc-clnt.c:938
#9  0x00007f2beb0aeb38 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:546
#10 0x00007f2bdfc027ed in socket_event_poll_in (this=0x7f2bd81f51c0) at socket.c:2353
#11 0x00007f2bdfc03abd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7f2bd81f51c0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466
#12 0x00007f2beb358080 in event_dispatch_epoll_handler (data=0x7f2bd80c0dd0) at event-epoll.c:575
---Type <return> to continue, or q <return> to quit---
#13 event_dispatch_epoll_worker (data=0x7f2bd80c0dd0) at event-epoll.c:678
#14 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bd654c700) at pthread_create.c:301
#15 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7f2bd5041700 (LWP 28324)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00007f2bea3d05d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007f2bea3d04a7 in __pthread_mutex_lock (mutex=0x7f2bd80022f0) at pthread_mutex_lock.c:61
#3  0x00007f2beb313171 in inode_ref (inode=0x7f2bd546e3e8) at inode.c:545
#4  0x00007f2beb3251e7 in __fd_create (inode=0x7f2bd546e3e8, pid=0) at fd.c:636
#5  0x00007f2beb3252c5 in __fd_anonymous (inode=0x7f2bd546e3e8, flags=2) at fd.c:782
#6  0x00007f2beb3253aa in fd_anonymous (inode=0x7f2bd546e3e8) at fd.c:806
#7  0x00007f2bdc857118 in ob_readv (frame=0x7f2be8ea8054, this=0x7f2bd801a8a0, fd=<value optimized out>, size=131072, offset=<value optimized out>, flags=<value optimized out>, xdata=0x0) at open-behind.c:381
#8  0x00007f2bdc642213 in mdc_readv (frame=0x7f2be8ea37c4, this=0x7f2bd801bc50, fd=0x7f2bd81f7b70, size=131072, offset=31571968, flags=32768, xdata=0x0) at md-cache.c:1550
#9  0x00007f2bdc426706 in io_stats_readv (frame=0x7f2be8ea799c, this=0x7f2bd801d060, fd=0x7f2bd81f7b70, size=131072, offset=31571968, flags=32768, xdata=0x0) at io-stats.c:2141
#10 0x00007f2beb2eed2f in default_readv (frame=0x7f2be8ea799c, this=0x7f2bd801e5a0, fd=0x7f2bd81f7b70, size=131072, offset=31571968, flags=32768, xdata=0x0) at defaults.c:1846
#11 0x00007f2bdc20f4ec in meta_readv (frame=0x7f2be8ea799c, this=0x7f2bd801e5a0, fd=0x7f2bd81f7b70, size=131072, offset=<value optimized out>, flags=<value optimized out>, xdata=0x0) at meta.c:79
#12 0x00007f2be2624d30 in fuse_readv_resume (state=<value optimized out>) at fuse-bridge.c:2242
#13 0x00007f2be261db65 in fuse_resolve_done (state=<value optimized out>) at fuse-resolve.c:668
#14 fuse_resolve_all (state=<value optimized out>) at fuse-resolve.c:695
#15 0x00007f2be261d896 in fuse_resolve (state=0x7f2bc801cbe0) at fuse-resolve.c:659
#16 0x00007f2be261dbae in fuse_resolve_all (state=<value optimized out>) at fuse-resolve.c:691
#17 0x00007f2be261dc13 in fuse_resolve_continue (state=<value optimized out>) at fuse-resolve.c:711
#18 0x00007f2be261d83e in fuse_resolve_fd (state=0x7f2bc801cbe0) at fuse-resolve.c:571
#19 fuse_resolve (state=0x7f2bc801cbe0) at fuse-resolve.c:648
#20 0x00007f2be261db8e in fuse_resolve_all (state=<value optimized out>) at fuse-resolve.c:684
#21 0x00007f2be261dbd8 in fuse_resolve_and_resume (state=0x7f2bc801cbe0, fn=0x7f2be2624aa0 <fuse_readv_resume>) at fuse-resolve.c:723
#22 0x00007f2be2633c70 in fuse_thread_proc (data=0x7f2bed9cb760) at fuse-bridge.c:4986
#23 0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bd5041700) at pthread_create.c:301
#24 0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7f2bcffff700 (LWP 28325)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f2be26269db in notify_kernel_loop (data=<value optimized out>) at fuse-bridge.c:3873
#2  0x00007f2bea3ceaa1 in start_thread (arg=0x7f2bcffff700) at pthread_create.c:301
#3  0x00007f2be9d37aad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f2beb7a5740 (LWP 28314)):
#0  0x00007f2bea3cf2fd in pthread_join (threadid=139826391009024, thread_return=0x0) at pthread_join.c:89
#1  0x00007f2beb357b2d in event_dispatch_epoll (event_pool=0x7f2bed9bcd30) at event-epoll.c:762
#2  0x00007f2beb7c5181 in main (argc=4, argv=0x7fffccaa5438) at glusterfsd.c:2370





Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 3 Raghavendra G 2017-06-13 03:29:25 EDT
Two mount process were deadlocked and one of them had active inodelks on a brick. Both the deadlocks involved interaction between open-behind and itable dump code

Deadlock1:

statedump had acquired locks in the order: itable -> inode -> fd
openbehind was trying to acquire locks in the order: fd -> itable (as part of inode_ref done after acquiring fd->lock)

Deadlock2:
statedump was trying to acquire lock: itable -> inode
openbehind was trying to acquire lock: inode -> itable (it was trying to create an anonymous fd which involves doing an inode_ref  after acquiring inode->lock)

Both the deadlocks are present in upstream master.

Note You need to log in before you can comment on or make changes to this bug.