| Summary: | [glusterfs-3.1.6qa6]: glusterfs client crashed in mem_put | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra Bhat <rabhat> |
| Component: | core | Assignee: | Amar Tumballi <amarts> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | pre-release | CC: | amarts, gluster-bugs, jdarcy, pkarampu, rahulcs, vijaykumar, vraman |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | --- | Mount Type: | fuse |
| Documentation: | --- | CRM: | |
| Verified Versions: | 3.1.7qa3 | Category: | --- |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Steps to reproduce: on glusterfs-3.1.6qa6 run posix compliance test and volume set option (which results in graph change) frequently parallely. CHANGE: http://review.gluster.com/194 (This can happen in deep_resolve_cbk, which happens after a graph) merged in release-3.2 by Anand Avati (avati) CHANGE: http://review.gluster.com/195 (This can happen in deep_resolve_cbk, which happens after a graph) merged in master by Anand Avati (avati) CHANGE: http://review.gluster.com/193 (This can happen in deep_resolve_cbk, which happens after a graph) merged in release-3.1 by Anand Avati (avati) patch sent to all the branches. This bug got exposed by preventing memory leak due to extra refs which we used to keep earlier. I tried running posix_compliant.sh on the mount point and volume set option simultaneously in release-3.1.6qa6, it got crashed and exited with signal 6. When i ran it on latest git pull, it ran successfully without any error. Its working fine. No crashes found with 3.1.7qa3 with simultaneous operations of ./configure, posix compliance & stat-prefetch on/offs. |
glusterfs client (fuse) crashed due to SIGABORT while putting the dentry back to the dentry pool. This is the backtrace of the core generated. Core was generated by `/usr/local/sbin/glusterfs --log-level=NORMAL --volfile-id=mirror --volfile-serv'. Program terminated with signal 6, Aborted. #0 0x000000350b830265 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x000000350b830265 in raise () from /lib64/libc.so.6 #1 0x000000350b831d10 in abort () from /lib64/libc.so.6 #2 0x000000350b86a99b in __libc_message () from /lib64/libc.so.6 #3 0x000000350b87245f in _int_free () from /lib64/libc.so.6 #4 0x000000350b8728bb in free () from /lib64/libc.so.6 #5 0x00002b409aafcc9c in mem_put (pool=0xe3a7e20, ptr=0x2aaaaca88128) at ../../../libglusterfs/src/mem-pool.c:510 #6 0x00002b409aae4ddd in __dentry_unset (dentry=0x2aaaaca88128) at ../../../libglusterfs/src/inode.c:149 #7 0x00002b409aae550a in __inode_retire (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:364 #8 0x00002b409aae55df in __inode_unref (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:388 #9 0x00002b409aae5695 in inode_unref (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:423 #10 0x00002b409aad028b in loc_wipe (loc=0xe3a8938) at ../../../libglusterfs/src/xlator.c:1086 #11 0x00002aaaab40de6a in afr_local_cleanup (local=0xe3a8900, this=0xe39ae90) at ../../../../../xlators/cluster/afr/src/afr-common.c:373 #12 0x00002aaaab3d384b in afr_unlink_done (frame=0x2b409bc62334, this=0xe39ae90) at ../../../../../xlators/cluster/afr/src/afr-dir-write.c:1717 #13 0x00002aaaab40491e in afr_unlock_common_cbk (frame=0x2b409bc62334, cookie=0x0, this=0xe39ae90, op_ret=0, op_errno=0) at ../../../../../xlators/cluster/afr/src/afr-lk-common.c:549 #14 0x00002aaaab4051a0 in afr_unlock_entrylk_cbk (frame=0x2b409bc62334, cookie=0x0, this=0xe39ae90, op_ret=0, op_errno=0) at ../../../../../xlators/cluster/afr/src/afr-lk-common.c:657 #15 0x00002aaaab1a0b2d in client3_1_entrylk_cbk (req=0x2aaaad67f044, iov=0x2aaaad67f084, count=1, myframe=0x2b409b9d2c74) at ../../../../../xlators/protocol/client/src/client3_1-fops.c:1132 #16 0x00002b409ad42505 in rpc_clnt_handle_reply (clnt=0xe3a6a00, pollin=0xe38af60) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:757 #17 0x00002b409ad4283c in rpc_clnt_notify (trans=0xe3a6bb0, mydata=0xe3a6a30, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe38af60) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:870 #18 0x00002b409ad3ffc9 in rpc_transport_notify (this=0xe3a6bb0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe38af60) at ../../../../rpc/rpc-lib/src/rpc-transport.c:1043 #19 0x00002aaaaad6c4bf in socket_event_poll_in (this=0xe3a6bb0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1623 #20 0x00002aaaaad6c830 in socket_event_handler (fd=11, idx=3, data=0xe3a6bb0, poll_in=1, poll_out=0, poll_err=0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1737 #21 0x00002b409aafb730 in event_dispatch_epoll_handler (event_pool=0xe369ba0, events=0xe36e5a0, i=1) at ../../../libglusterfs/src/event.c:812 #22 0x00002b409aafb91f in event_dispatch_epoll (event_pool=0xe369ba0) at ../../../libglusterfs/src/event.c:876 #23 0x00002b409aafbc7b in event_dispatch (event_pool=0xe369ba0) at ../../../libglusterfs/src/event.c:984 #24 0x0000000000405fc7 in main (argc=5, argv=0x7fff5fd02968) at ../../../glusterfsd/src/glusterfsd.c:1455 (gdb) p *(dentry_t *)ptr $1 = {inode_list = {next = 0x2aaaaca88128, prev = 0x2aaaaca88128}, hash = {next = 0x2aaaaca88138, prev = 0x2aaaaca88138}, inode = 0x2aaaad939488, name = 0xe3a82a0 "fstest_e6380807a78ebfccf3c10f2dc836f3cf", parent = 0x0} (gdb) f 8 #8 0x00002b409aae55df in __inode_unref (inode=0x2aaaad939488) at ../../../libglusterfs/src/inode.c:388 388 __inode_retire (inode); (gdb) p *inode $2 = {table = 0xe3a7cc0, gfid = "\314\361\267\371ǘAh\266@\332_\350Y\255\005", lock = 1, nlookup = 0, ref = 0, ino = 4314840, ia_type = IA_IFREG, fd_list = {next = 0x2aaaad9394c8, prev = 0x2aaaad9394c8}, dentry_list = {next = 0x2aaaad9394d8, prev = 0x2aaaad9394d8}, hash = {next = 0x2aaaad9394e8, prev = 0x2aaaad9394e8}, list = {next = 0xe3a7d50, prev = 0xe3a7d50}, _ctx = 0xe3ab000} (gdb) Seems to be a double free.