Bug 765087 (GLUSTER-3355) - [glusterfs-3.1.6qa6]: glusterfs client crashed in mem_put
Summary: [glusterfs-3.1.6qa6]: glusterfs client crashed in mem_put
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: GLUSTER-3355
Product: GlusterFS
Classification: Community
Component: core
Version: pre-release
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
Assignee: Amar Tumballi
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-08 05:36 UTC by Raghavendra Bhat
Modified: 2013-12-19 00:06 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression: ---
Mount Type: fuse
Documentation: ---
CRM:
Verified Versions: 3.1.7qa3


Attachments (Terms of Use)

Description Raghavendra Bhat 2011-08-08 05:36:55 UTC
glusterfs client (fuse) crashed due to SIGABORT while putting the dentry back to the dentry pool. This is the backtrace of the core generated.



Core was generated by `/usr/local/sbin/glusterfs --log-level=NORMAL --volfile-id=mirror --volfile-serv'.
Program terminated with signal 6, Aborted.
#0  0x000000350b830265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x000000350b830265 in raise () from /lib64/libc.so.6
#1  0x000000350b831d10 in abort () from /lib64/libc.so.6
#2  0x000000350b86a99b in __libc_message () from /lib64/libc.so.6
#3  0x000000350b87245f in _int_free () from /lib64/libc.so.6
#4  0x000000350b8728bb in free () from /lib64/libc.so.6
#5  0x00002b409aafcc9c in mem_put (pool=0xe3a7e20, ptr=0x2aaaaca88128) at ../../../libglusterfs/src/mem-pool.c:510
#6  0x00002b409aae4ddd in __dentry_unset (dentry=0x2aaaaca88128) at ../../../libglusterfs/src/inode.c:149
#7  0x00002b409aae550a in __inode_retire (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:364
#8  0x00002b409aae55df in __inode_unref (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:388
#9  0x00002b409aae5695 in inode_unref (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:423
#10 0x00002b409aad028b in loc_wipe (loc=0xe3a8938) at ../../../libglusterfs/src/xlator.c:1086
#11 0x00002aaaab40de6a in afr_local_cleanup (local=0xe3a8900, this=0xe39ae90) at ../../../../../xlators/cluster/afr/src/afr-common.c:373
#12 0x00002aaaab3d384b in afr_unlink_done (frame=0x2b409bc62334, this=0xe39ae90)
    at ../../../../../xlators/cluster/afr/src/afr-dir-write.c:1717
#13 0x00002aaaab40491e in afr_unlock_common_cbk (frame=0x2b409bc62334, cookie=0x0, this=0xe39ae90, op_ret=0, op_errno=0)
    at ../../../../../xlators/cluster/afr/src/afr-lk-common.c:549
#14 0x00002aaaab4051a0 in afr_unlock_entrylk_cbk (frame=0x2b409bc62334, cookie=0x0, this=0xe39ae90, op_ret=0, op_errno=0)
    at ../../../../../xlators/cluster/afr/src/afr-lk-common.c:657
#15 0x00002aaaab1a0b2d in client3_1_entrylk_cbk (req=0x2aaaad67f044, iov=0x2aaaad67f084, count=1, myframe=0x2b409b9d2c74)
    at ../../../../../xlators/protocol/client/src/client3_1-fops.c:1132
#16 0x00002b409ad42505 in rpc_clnt_handle_reply (clnt=0xe3a6a00, pollin=0xe38af60) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:757
#17 0x00002b409ad4283c in rpc_clnt_notify (trans=0xe3a6bb0, mydata=0xe3a6a30, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe38af60)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:870
#18 0x00002b409ad3ffc9 in rpc_transport_notify (this=0xe3a6bb0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe38af60)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:1043
#19 0x00002aaaaad6c4bf in socket_event_poll_in (this=0xe3a6bb0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1623
#20 0x00002aaaaad6c830 in socket_event_handler (fd=11, idx=3, data=0xe3a6bb0, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1737
#21 0x00002b409aafb730 in event_dispatch_epoll_handler (event_pool=0xe369ba0, events=0xe36e5a0, i=1)
    at ../../../libglusterfs/src/event.c:812
#22 0x00002b409aafb91f in event_dispatch_epoll (event_pool=0xe369ba0) at ../../../libglusterfs/src/event.c:876
#23 0x00002b409aafbc7b in event_dispatch (event_pool=0xe369ba0) at ../../../libglusterfs/src/event.c:984
#24 0x0000000000405fc7 in main (argc=5, argv=0x7fff5fd02968) at ../../../glusterfsd/src/glusterfsd.c:1455
(gdb) p *(dentry_t *)ptr
$1 = {inode_list = {next = 0x2aaaaca88128, prev = 0x2aaaaca88128}, hash = {next = 0x2aaaaca88138, prev = 0x2aaaaca88138}, 
  inode = 0x2aaaad939488, name = 0xe3a82a0 "fstest_e6380807a78ebfccf3c10f2dc836f3cf", parent = 0x0}
(gdb) f 8
#8  0x00002b409aae55df in __inode_unref (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:388
388                             __inode_retire (inode);
(gdb) p *inode
$2 = {table = 0xe3a7cc0, gfid = "\314\361\267\371ǘAh\266@\332_\350Y\255\005", lock = 1, nlookup = 0, ref = 0, ino = 4314840, 
  ia_type = IA_IFREG, fd_list = {next = 0x2aaaad9394c8, prev = 0x2aaaad9394c8}, dentry_list = {next = 0x2aaaad9394d8, 
    prev = 0x2aaaad9394d8}, hash = {next = 0x2aaaad9394e8, prev = 0x2aaaad9394e8}, list = {next = 0xe3a7d50, prev = 0xe3a7d50}, 
  _ctx = 0xe3ab000}
(gdb) 


Seems to be a double free.

Comment 1 Pranith Kumar K 2011-08-09 03:18:54 UTC
Steps to reproduce:
on glusterfs-3.1.6qa6 run posix compliance test and volume set option (which results in graph change) frequently parallely.

Comment 2 Anand Avati 2011-08-10 10:50:59 UTC
CHANGE: http://review.gluster.com/194 (This can happen in deep_resolve_cbk, which happens after a graph) merged in release-3.2 by Anand Avati (avati)

Comment 3 Anand Avati 2011-08-10 10:53:08 UTC
CHANGE: http://review.gluster.com/195 (This can happen in deep_resolve_cbk, which happens after a graph) merged in master by Anand Avati (avati)

Comment 4 Anand Avati 2011-08-10 10:58:50 UTC
CHANGE: http://review.gluster.com/193 (This can happen in deep_resolve_cbk, which happens after a graph) merged in release-3.1 by Anand Avati (avati)

Comment 5 Amar Tumballi 2011-08-10 23:48:48 UTC
patch sent to all the branches.

This bug got exposed by preventing memory leak due to extra refs which we used to keep earlier.

Comment 6 Vijaykumar 2011-08-19 07:29:31 UTC
I tried running posix_compliant.sh on the mount point and volume set option simultaneously in release-3.1.6qa6, it got crashed and exited with signal 6. When i ran it on latest git pull, it ran successfully without any error. Its working fine.

Comment 7 Rahul C S 2011-09-05 06:27:52 UTC
No crashes found with 3.1.7qa3 with simultaneous operations of ./configure, posix compliance & stat-prefetch on/offs.


Note You need to log in before you can comment on or make changes to this bug.