Description of problem: Created a distributed-replicate volume, on the fuse client was running dbench. The client got crashed after sometime at mdc_lookup. so remounted the client. From a nfs client, start doing rmdir on the dbench created directories. This caused a crash on the nfs server. Core backtrace: Core was generated by `/usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/'. Program terminated with signal 6, Aborted. #0 0x0000003ae5632a45 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6.x86_64 libgcc-4.4.5-6.el6.x86_64 (gdb) bt #0 0x0000003ae5632a45 in raise () from /lib64/libc.so.6 #1 0x0000003ae5634225 in abort () from /lib64/libc.so.6 #2 0x0000003ae562b9d5 in __assert_fail () from /lib64/libc.so.6 #3 0x00007f1d7883d188 in client3_1_opendir (frame=0x7f1d7ab99678, this=0xc535a0, data=0x7fff6bf54340) at client3_1-fops.c:3741 #4 0x00007f1d78824aed in client_opendir (frame=0x7f1d7ab99678, this=0xc535a0, loc=0x7f1d75919484, fd=0xc64480) at client.c:1027 #5 0x00007f1d7859d9cd in afr_opendir (frame=0x7f1d7ab9a33c, this=0xc56a80, loc=0x7f1d75919484, fd=0xc64480) at afr-dir-read.c:335 #6 0x00007f1d7836e402 in dht_rmdir (frame=0x7f1d7ab99bd8, this=0xc589a0, loc=0x7f1d75919484, flags=0) at dht-common.c:4248 #7 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9a290, this=0xc59cc0, loc=0x7f1d75919484, flags=0) at defaults.c:969 #8 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9a08c, this=0xc5b120, loc=0x7f1d75919484, flags=0) at defaults.c:969 #9 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9b000, this=0xc5c400, loc=0x7f1d75919484, flags=0) at defaults.c:969 #10 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab98e68, this=0xc5d610, loc=0x7f1d75919484, flags=0) at defaults.c:969 #11 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab98b0c, this=0xc5e860, loc=0x7f1d75919484, flags=0) at defaults.c:969 #12 0x00007f1d776bda1e in io_stats_rmdir (frame=0x7f1d7ab9a540, this=0xc5fb10, loc=0x7f1d75919484, flags=0) at io-stats.c:1944 #13 0x00007f1d77468222 in nfs_fop_rmdir (nfsx=0xc60ef0, xl=0xc5fb10, nfu=0x7fff6bf54b70, pathloc=0x7f1d75919484, cbk=0x7f1d7746c5fe <nfs_inode_rmdir_cbk>, local=0x7f1d7742c2c0) at nfs-fops.c:998 #14 0x00007f1d7746c84f in nfs_inode_rmdir (nfsx=0xc60ef0, xl=0xc5fb10, nfu=0x7fff6bf54b70, pathloc=0x7f1d75919484, cbk=0x7f1d77480bed <nfs3svc_rmdir_cbk>, local=0x7f1d7591904c) at nfs-inodes.c:446 #15 0x00007f1d7746d913 in nfs_rmdir (nfsx=0xc60ef0, xl=0xc5fb10, nfu=0x7fff6bf54b70, path=0x7f1d75919484, cbk=0x7f1d77480bed <nfs3svc_rmdir_cbk>, local=0x7f1d7591904c) at nfs-generics.c:252 #16 0x00007f1d77480ece in nfs3_rmdir_resume (carg=0x7f1d7591904c) at nfs3.c:3539 #17 0x00007f1d774908ad in nfs3_fh_resolve_entry_lookup_cbk (frame=0x7f1d7a993924, cookie=0xc5fb10, this=0xc60ef0, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at nfs3-helpers.c:3633 #18 0x00007f1d77462d87 in nfs_fop_lookup_cbk (frame=0x7f1d7a993924, cookie=0xc5fb10, this=0xc60ef0, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at nfs-fops.c:316 #19 0x00007f1d776b7c4a in io_stats_lookup_cbk (frame=0x7f1d7ab9af54, cookie=0x7f1d7ab9a89c, this=0xc5fb10, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at io-stats.c:1468 #20 0x00007f1d778cdace in qr_lookup_cbk (frame=0x7f1d7ab9a89c, cookie=0x7f1d7ab98bb8, this=0xc5e860, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at quick-read.c:417 #21 0x00007f1d77ae88cc in ioc_lookup_cbk (frame=0x7f1d7ab98bb8, cookie=0x7f1d7ab9a744, this=0xc5d610, op_ret=0, op_errno=0, inode=0x7f1d7645e298, stbuf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at io-cache.c:266 #22 0x00007f1d7bd6475a in default_lookup_cbk (frame=0x7f1d7ab9a744, cookie=0x7f1d7ab989b4, this=0xc5c400, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at defaults.c:46 #23 0x00007f1d7bd6475a in default_lookup_cbk (frame=0x7f1d7ab989b4, cookie=0x7f1d7ab9a3e8, this=0xc5b120, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at defaults.c:46 #24 0x00007f1d7811f2f6 in quota_lookup_cbk (frame=0x7f1d7ab9a3e8, cookie=0x7f1d7ab99270, this=0xc59cc0, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at quota.c:666 #25 0x00007f1d7835851a in dht_lookup_cbk (frame=0x7f1d7ab99270, cookie=0x7f1d7ab98f14, this=0xc589a0, op_ret=0, op_errno=0, inode=0x7f1d7645e298, stbuf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at dht-common.c:1266 #26 0x00007f1d785f37b2 in afr_lookup_done (frame=0x7f1d7ab98f14, this=0xc56a80) at afr-common.c:1743 #27 0x00007f1d785f3eb4 in afr_lookup_cbk (frame=0x7f1d7ab98f14, cookie=0x1, this=0xc56a80, op_ret=0, op_errno=0, inode=0x7f1d7645e298, buf=0x7fff6bf55870, xattr=0xd9d420, postparent=0x7fff6bf55800) at afr-common.c:1906 #28 0x00007f1d78836347 in client3_1_lookup_cbk (req=0x7f1d740da908, iov=0x7f1d740da948, count=1, myframe=0x7f1d7ab98250) at client3_1-fops.c:2185 #29 0x00007f1d7bb3d919 in rpc_clnt_handle_reply (clnt=0xcfd2d0, pollin=0xd9da20) at rpc-clnt.c:796 #30 0x00007f1d7bb3dcb6 in rpc_clnt_notify (trans=0xd0ce50, mydata=0xcfd300, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xd9da20) at rpc-clnt.c:915 #31 0x00007f1d7bb39da8 in rpc_transport_notify (this=0xd0ce50, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xd9da20) at rpc-transport.c:498 #32 0x00007f1d759e2270 in socket_event_poll_in (this=0xd0ce50) at socket.c:1686 #33 0x00007f1d759e27f4 in socket_event_handler (fd=13, idx=5, data=0xd0ce50, poll_in=1, poll_out=0, poll_err=0) at socket.c:1801 #34 0x00007f1d7bd9405c in event_dispatch_epoll_handler (event_pool=0xc48c20, events=0xd33570, i=0) at event.c:794 #35 0x00007f1d7bd9427f in event_dispatch_epoll (event_pool=0xc48c20) at event.c:856 #36 0x00007f1d7bd9460a in event_dispatch (event_pool=0xc48c20) at event.c:956 #37 0x0000000000407dcc in main (argc=7, argv=0x7fff6bf55f68) at glusterfsd.c:1612 (gdb) f 8 #8 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9a08c, this=0xc5b120, loc=0x7f1d75919484, flags=0) at defaults.c:969 969 STACK_WIND (frame, default_rmdir_cbk, FIRST_CHILD(this), (gdb) l 964 } 965 966 int32_t 967 default_rmdir (call_frame_t *frame, xlator_t *this, loc_t *loc, int flags) 968 { 969 STACK_WIND (frame, default_rmdir_cbk, FIRST_CHILD(this), 970 FIRST_CHILD(this)->fops->rmdir, loc, flags); 971 return 0; 972 } 973 (gdb) p *loc $1 = {path = 0xd98aa0 "/clients/client21/~dmtmp", name = 0xd98ab2 "~dmtmp", inode = 0x7f1d7645e298, parent = 0x7f1d7645d674, gfid = '\000' <repeats 15 times>, pargfid = "\025M\362\213\224\317NY\262\021\326\026]\r\316", <incomplete sequence \347>} (gdb) f 3 #3 0x00007f1d7883d188 in client3_1_opendir (frame=0x7f1d7ab99678, this=0xc535a0, data=0x7fff6bf54340) at client3_1-fops.c:3741 3741 GF_ASSERT_AND_GOTO_WITH_ERROR (this->name, (gdb) l 3736 if (!uuid_is_null (args->loc->inode->gfid)) 3737 memcpy (req.gfid, args->loc->inode->gfid, 16); 3738 else 3739 memcpy (req.gfid, args->loc->gfid, 16); 3740 3741 GF_ASSERT_AND_GOTO_WITH_ERROR (this->name, 3742 !uuid_is_null (*((uuid_t*)req.gfid)), 3743 unwind, op_errno, EINVAL); 3744 3745 conf = this->private; Nfs log: [2012-02-29 05:19:56.802181] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:80be6e28-0b2f-4920-b6b4-6c072 b08d586) [2012-02-29 05:19:56.804903] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:80be6e28-0b2f-4920-b6b4-6c072 b08d586) [2012-02-29 05:19:56.807738] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7 d38538), hence aborting size updation of parents [2012-02-29 05:19:56.822073] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:f44ba045-5294-429f-a7da-30223 5ed6854) [2012-02-29 05:19:56.826532] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:f44ba045-5294-429f-a7da-30223 5ed6854) [2012-02-29 05:19:56.831382] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7 d38538), hence aborting size updation of parents [2012-02-29 05:19:56.841114] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7d38538), hence aborting size updation of parents [2012-02-29 05:19:56.851085] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7d38538), hence aborting size updation of parents [2012-02-29 05:19:56.857425] W [dict.c:461:dict_ref] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(+0x6559d) [0x7f1d785f359d] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_lookup_done_success_action+0x11b) [0x7f1d785f32e1] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_lookup_build_response_params+0x140) [0x7f1d785f11bf]))) 0-dict: dict is NULL pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 6 time of crash: 2012-02-29 05:19:56 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3git /lib64/libc.so.6[0x3ae5632ac0] /lib64/libc.so.6(gsignal+0x35)[0x3ae5632a45] /lib64/libc.so.6(abort+0x175)[0x3ae5634225] /lib64/libc.so.6(__assert_fail+0xf5)[0x3ae562b9d5] /usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_opendir+0x1a9)[0x7f1d7883d188] /usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client_opendir+0x163)[0x7f1d78824aed] /usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_opendir+0x5e4)[0x7f1d7859d9cd] /usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_rmdir+0x664)[0x7f1d7836e402] /usr/local/lib/libglusterfs.so.0(default_rmdir+0x22e)[0x7f1d7bd70b1d] ---------
*** This bug has been marked as a duplicate of bug 773225 ***