Bug 762708 (GLUSTER-976)

Summary: Deadlock in state dumping while holding the logging lock
Product: [Community] GlusterFS Reporter: Shehjar Tikoo <shehjart>
Component: coreAssignee: kaushik <kbudiger>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: nfs-alphaCC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Shehjar Tikoo 2010-06-02 10:35:21 UTC
Heres the backtrace:

Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libnss_mdns4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_mdns4.so.2
0x00007f1a59890464 in __lll_lock_wait () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007f1a59890464 in __lll_lock_wait () from /lib/libpthread.so.0
#1  0x00007f1a5988b5d9 in _L_lock_953 () from /lib/libpthread.so.0
#2  0x00007f1a5988b3fb in pthread_mutex_lock () from /lib/libpthread.so.0
#3  0x00007f1a59cbfabb in _gf_log (domain=0x259c010 "posix-locked", file=0x7f1a584a10f6 "inodelk.c", function=0x7f1a584a1480 "__get_inodelk_count", line=691, level=GF_LOG_DEBUG, 
    fmt=0x7f1a584a13d8 " XATTR DEBUG domain: %s %s (pid=%d) (lk-owner=%lu) %ld - %ld state = Active") at logging.c:482
#4  0x00007f1a5849fcba in __get_inodelk_count (this=0x259c030, pl_inode=0x25ab780) at inodelk.c:682
#5  0x00007f1a5849fe6c in get_inodelk_count (this=0x259c030, inode=0x25ab8d0) at inodelk.c:734
#6  0x00007f1a5849b092 in pl_dump_inode_priv (this=0x259c030, inode=0x25ab8d0) at posix.c:1400
#7  0x00007f1a59cd1d62 in inode_dump (inode=0x25ab8d0, prefix=0x7fff7c58dba0 "xlator.protocol.server.conn.1.bound_xl.posix-locked-iot.active.1") at inode.c:1373
#8  0x00007f1a59cd20d4 in inode_table_dump (itable=0x25a3cb0, prefix=0x7fff7c58ec10 "xlator.protocol.server.conn.1.bound_xl.posix-locked-iot") at inode.c:1414
#9  0x00007f1a5806ce64 in server_inode (this=0x259e280) at server-protocol.c:6169
#10 0x00007f1a59cea2ec in gf_proc_dump_xlator_info (this_xl=0x259e280) at statedump.c:163
#11 0x00007f1a59cea3ac in gf_proc_dump_info (signum=10) at statedump.c:189
#12 <signal handler called>
#13 0x00007f1a595d84dd in write () from /lib/libc.so.6
#14 0x00007f1a59573133 in _IO_file_write () from /lib/libc.so.6
#15 0x00007f1a59574785 in _IO_do_write () from /lib/libc.so.6
#16 0x00007f1a595738e8 in _IO_file_sync () from /lib/libc.so.6
#17 0x00007f1a59567f9a in fflush () from /lib/libc.so.6
#18 0x00007f1a59cbfcba in _gf_log (domain=0x259f6a0 "repl", file=0x7f1a57e47f3d "afr-self-heal-algorithm.c", function=0x7f1a57e48300 "sh_full_write_cbk", line=143, 
    level=GF_LOG_TRACE, fmt=0x7f1a57e47f58 "wrote %d bytes of data from %s to child %d, offset %ld") at logging.c:516
#19 0x00007f1a57e4136d in sh_full_write_cbk (rw_frame=0x27cf588, cookie=0x1, this=0x259f6c0, op_ret=65536, op_errno=0, prebuf=0x7fff7c590540, postbuf=0x7fff7c5904d0)
    at afr-self-heal-algorithm.c:140
#20 0x00007f1a58af0eab in client_write_cbk (frame=0x267f850, hdr=0x27cf690, hdrlen=268, iobuf=0x0) at client-protocol.c:4196
#21 0x00007f1a58af6de6 in protocol_client_interpret (this=0x2598f30, trans=0x25a5960, hdr_p=0x27cf690 "", hdrlen=268, iobuf=0x0) at client-protocol.c:6093
#22 0x00007f1a58af7b67 in protocol_client_pollin (this=0x2598f30, trans=0x25a5960) at client-protocol.c:6391
#23 0x00007f1a58af8160 in notify (this=0x2598f30, event=2, data=0x25a5960) at client-protocol.c:6510
#24 0x00007f1a59cbe1c8 in xlator_notify (xl=0x2598f30, event=2, data=0x25a5960) at xlator.c:919
#25 0x00007f1a56c9746a in socket_event_poll_in (this=0x25a5960) at socket.c:731
#26 0x00007f1a56c9779d in socket_event_handler (fd=10, idx=5, data=0x25a5960, poll_in=1, poll_out=0, poll_err=0) at socket.c:831
#27 0x00007f1a59ce572e in event_dispatch_epoll_handler (event_pool=0x2592330, events=0x25a8350, i=0) at event.c:804
#28 0x00007f1a59ce5920 in event_dispatch_epoll (event_pool=0x2592330) at event.c:867
#29 0x00007f1a59ce5c3f in event_dispatch (event_pool=0x2592330) at event.c:975
#30 0x0000000000406cc0 in main (argc=8, argv=0x7fff7c5911d8) at glusterfsd.c:1494
(gdb)

Comment 1 Vijay Bellur 2010-09-15 09:09:25 UTC
PATCH: http://patches.gluster.com/patch/4570 in master (Deadlock in state dumping while holding the logging lock)