Bug 798608 - [c2c82c360fda20533093c27c82ce0725cf40cab5]: nfs server crashed at client3_1_opendir because gfid is null
[c2c82c360fda20533093c27c82ce0725cf40cab5]: nfs server crashed at client3_1_o...
Status: CLOSED DUPLICATE of bug 773225
Product: GlusterFS
Classification: Community
Component: core (Show other bugs)
pre-release
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Pranith Kumar K
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-02-29 06:27 EST by Rahul C S
Modified: 2012-03-04 01:44 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-04 01:44:16 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul C S 2012-02-29 06:27:22 EST
Description of problem:
Created a distributed-replicate volume, on the fuse client was running dbench. The client got crashed after sometime at mdc_lookup. so remounted the client. From a nfs client, start doing rmdir on the dbench created directories. This caused a crash on the nfs server.


Core backtrace:
Core was generated by `/usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/'.
Program terminated with signal 6, Aborted.
#0  0x0000003ae5632a45 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6.x86_64 libgcc-4.4.5-6.el6.x86_64
(gdb) bt
#0  0x0000003ae5632a45 in raise () from /lib64/libc.so.6
#1  0x0000003ae5634225 in abort () from /lib64/libc.so.6
#2  0x0000003ae562b9d5 in __assert_fail () from /lib64/libc.so.6
#3  0x00007f1d7883d188 in client3_1_opendir (frame=0x7f1d7ab99678, this=0xc535a0, data=0x7fff6bf54340) at client3_1-fops.c:3741
#4  0x00007f1d78824aed in client_opendir (frame=0x7f1d7ab99678, this=0xc535a0, loc=0x7f1d75919484, fd=0xc64480) at client.c:1027
#5  0x00007f1d7859d9cd in afr_opendir (frame=0x7f1d7ab9a33c, this=0xc56a80, loc=0x7f1d75919484, fd=0xc64480) at afr-dir-read.c:335
#6  0x00007f1d7836e402 in dht_rmdir (frame=0x7f1d7ab99bd8, this=0xc589a0, loc=0x7f1d75919484, flags=0) at dht-common.c:4248
#7  0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9a290, this=0xc59cc0, loc=0x7f1d75919484, flags=0) at defaults.c:969
#8  0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9a08c, this=0xc5b120, loc=0x7f1d75919484, flags=0) at defaults.c:969
#9  0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9b000, this=0xc5c400, loc=0x7f1d75919484, flags=0) at defaults.c:969
#10 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab98e68, this=0xc5d610, loc=0x7f1d75919484, flags=0) at defaults.c:969
#11 0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab98b0c, this=0xc5e860, loc=0x7f1d75919484, flags=0) at defaults.c:969
#12 0x00007f1d776bda1e in io_stats_rmdir (frame=0x7f1d7ab9a540, this=0xc5fb10, loc=0x7f1d75919484, flags=0) at io-stats.c:1944
#13 0x00007f1d77468222 in nfs_fop_rmdir (nfsx=0xc60ef0, xl=0xc5fb10, nfu=0x7fff6bf54b70, pathloc=0x7f1d75919484,
    cbk=0x7f1d7746c5fe <nfs_inode_rmdir_cbk>, local=0x7f1d7742c2c0) at nfs-fops.c:998
#14 0x00007f1d7746c84f in nfs_inode_rmdir (nfsx=0xc60ef0, xl=0xc5fb10, nfu=0x7fff6bf54b70, pathloc=0x7f1d75919484,
    cbk=0x7f1d77480bed <nfs3svc_rmdir_cbk>, local=0x7f1d7591904c) at nfs-inodes.c:446
#15 0x00007f1d7746d913 in nfs_rmdir (nfsx=0xc60ef0, xl=0xc5fb10, nfu=0x7fff6bf54b70, path=0x7f1d75919484,
    cbk=0x7f1d77480bed <nfs3svc_rmdir_cbk>, local=0x7f1d7591904c) at nfs-generics.c:252
#16 0x00007f1d77480ece in nfs3_rmdir_resume (carg=0x7f1d7591904c) at nfs3.c:3539
#17 0x00007f1d774908ad in nfs3_fh_resolve_entry_lookup_cbk (frame=0x7f1d7a993924, cookie=0xc5fb10, this=0xc60ef0, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at nfs3-helpers.c:3633
#18 0x00007f1d77462d87 in nfs_fop_lookup_cbk (frame=0x7f1d7a993924, cookie=0xc5fb10, this=0xc60ef0, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at nfs-fops.c:316
#19 0x00007f1d776b7c4a in io_stats_lookup_cbk (frame=0x7f1d7ab9af54, cookie=0x7f1d7ab9a89c, this=0xc5fb10, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at io-stats.c:1468
#20 0x00007f1d778cdace in qr_lookup_cbk (frame=0x7f1d7ab9a89c, cookie=0x7f1d7ab98bb8, this=0xc5e860, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at quick-read.c:417
#21 0x00007f1d77ae88cc in ioc_lookup_cbk (frame=0x7f1d7ab98bb8, cookie=0x7f1d7ab9a744, this=0xc5d610, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, stbuf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at io-cache.c:266
#22 0x00007f1d7bd6475a in default_lookup_cbk (frame=0x7f1d7ab9a744, cookie=0x7f1d7ab989b4, this=0xc5c400, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at defaults.c:46
#23 0x00007f1d7bd6475a in default_lookup_cbk (frame=0x7f1d7ab989b4, cookie=0x7f1d7ab9a3e8, this=0xc5b120, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at defaults.c:46
#24 0x00007f1d7811f2f6 in quota_lookup_cbk (frame=0x7f1d7ab9a3e8, cookie=0x7f1d7ab99270, this=0xc59cc0, op_ret=0, op_errno=0, 
    inode=0x7f1d7645e298, buf=0x7f1d6e1cb8e8, dict=0x0, postparent=0x7f1d6e1cb958) at quota.c:666
#25 0x00007f1d7835851a in dht_lookup_cbk (frame=0x7f1d7ab99270, cookie=0x7f1d7ab98f14, this=0xc589a0, op_ret=0, op_errno=0,
    inode=0x7f1d7645e298, stbuf=0x7f1d6e1cb8e8, xattr=0x0, postparent=0x7f1d6e1cb958) at dht-common.c:1266
#26 0x00007f1d785f37b2 in afr_lookup_done (frame=0x7f1d7ab98f14, this=0xc56a80) at afr-common.c:1743
#27 0x00007f1d785f3eb4 in afr_lookup_cbk (frame=0x7f1d7ab98f14, cookie=0x1, this=0xc56a80, op_ret=0, op_errno=0, inode=0x7f1d7645e298,
    buf=0x7fff6bf55870, xattr=0xd9d420, postparent=0x7fff6bf55800) at afr-common.c:1906
#28 0x00007f1d78836347 in client3_1_lookup_cbk (req=0x7f1d740da908, iov=0x7f1d740da948, count=1, myframe=0x7f1d7ab98250)
    at client3_1-fops.c:2185
#29 0x00007f1d7bb3d919 in rpc_clnt_handle_reply (clnt=0xcfd2d0, pollin=0xd9da20) at rpc-clnt.c:796
#30 0x00007f1d7bb3dcb6 in rpc_clnt_notify (trans=0xd0ce50, mydata=0xcfd300, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xd9da20)
    at rpc-clnt.c:915
#31 0x00007f1d7bb39da8 in rpc_transport_notify (this=0xd0ce50, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xd9da20) at rpc-transport.c:498
#32 0x00007f1d759e2270 in socket_event_poll_in (this=0xd0ce50) at socket.c:1686
#33 0x00007f1d759e27f4 in socket_event_handler (fd=13, idx=5, data=0xd0ce50, poll_in=1, poll_out=0, poll_err=0) at socket.c:1801
#34 0x00007f1d7bd9405c in event_dispatch_epoll_handler (event_pool=0xc48c20, events=0xd33570, i=0) at event.c:794
#35 0x00007f1d7bd9427f in event_dispatch_epoll (event_pool=0xc48c20) at event.c:856
#36 0x00007f1d7bd9460a in event_dispatch (event_pool=0xc48c20) at event.c:956
#37 0x0000000000407dcc in main (argc=7, argv=0x7fff6bf55f68) at glusterfsd.c:1612
(gdb) f 8
#8  0x00007f1d7bd70b1d in default_rmdir (frame=0x7f1d7ab9a08c, this=0xc5b120, loc=0x7f1d75919484, flags=0) at defaults.c:969
969             STACK_WIND (frame, default_rmdir_cbk, FIRST_CHILD(this),
(gdb) l
964     }
965
966     int32_t
967     default_rmdir (call_frame_t *frame, xlator_t *this, loc_t *loc, int flags)
968     {
969             STACK_WIND (frame, default_rmdir_cbk, FIRST_CHILD(this),
970                         FIRST_CHILD(this)->fops->rmdir, loc, flags);
971             return 0;
972     }
973
(gdb) p *loc
$1 = {path = 0xd98aa0 "/clients/client21/~dmtmp", name = 0xd98ab2 "~dmtmp", inode = 0x7f1d7645e298, parent = 0x7f1d7645d674, 
  gfid = '\000' <repeats 15 times>, pargfid = "\025M\362\213\224\317NY\262\021\326\026]\r\316", <incomplete sequence \347>}
(gdb) f 3
#3  0x00007f1d7883d188 in client3_1_opendir (frame=0x7f1d7ab99678, this=0xc535a0, data=0x7fff6bf54340) at client3_1-fops.c:3741
3741            GF_ASSERT_AND_GOTO_WITH_ERROR (this->name,
(gdb) l
3736            if (!uuid_is_null (args->loc->inode->gfid))
3737                    memcpy (req.gfid,  args->loc->inode->gfid, 16);
3738            else
3739                    memcpy (req.gfid, args->loc->gfid, 16);
3740
3741            GF_ASSERT_AND_GOTO_WITH_ERROR (this->name,
3742                                           !uuid_is_null (*((uuid_t*)req.gfid)),
3743                                           unwind, op_errno, EINVAL);
3744
3745            conf = this->private;


Nfs log:
[2012-02-29 05:19:56.802181] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:80be6e28-0b2f-4920-b6b4-6c072
b08d586)
[2012-02-29 05:19:56.804903] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:80be6e28-0b2f-4920-b6b4-6c072
b08d586)
[2012-02-29 05:19:56.807738] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7
d38538), hence aborting size updation of parents
[2012-02-29 05:19:56.822073] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:f44ba045-5294-429f-a7da-30223
5ed6854)
[2012-02-29 05:19:56.826532] W [quota.c:2146:quota_fstat_cbk] 0-vol-quota: quota context not set in inode (gfid:f44ba045-5294-429f-a7da-30223
5ed6854)
[2012-02-29 05:19:56.831382] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7
d38538), hence aborting size updation of parents
[2012-02-29 05:19:56.841114] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7d38538), hence aborting size updation of parents
[2012-02-29 05:19:56.851085] W [quota.c:779:quota_update_size] 0-vol-quota: cannot find parent for inode (gfid:198659da-067e-4812-8a26-36aec7d38538), hence aborting size updation of parents
[2012-02-29 05:19:56.857425] W [dict.c:461:dict_ref] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(+0x6559d) [0x7f1d785f359d] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_lookup_done_success_action+0x11b) [0x7f1d785f32e1] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_lookup_build_response_params+0x140) [0x7f1d785f11bf]))) 0-dict: dict is NULL
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 2012-02-29 05:19:56
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3git
/lib64/libc.so.6[0x3ae5632ac0]
/lib64/libc.so.6(gsignal+0x35)[0x3ae5632a45]
/lib64/libc.so.6(abort+0x175)[0x3ae5634225]
/lib64/libc.so.6(__assert_fail+0xf5)[0x3ae562b9d5]
/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_opendir+0x1a9)[0x7f1d7883d188]
/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client_opendir+0x163)[0x7f1d78824aed]
/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_opendir+0x5e4)[0x7f1d7859d9cd]
/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_rmdir+0x664)[0x7f1d7836e402]
/usr/local/lib/libglusterfs.so.0(default_rmdir+0x22e)[0x7f1d7bd70b1d]
---------
Comment 1 Pranith Kumar K 2012-03-04 01:44:16 EST

*** This bug has been marked as a duplicate of bug 773225 ***

Note You need to log in before you can comment on or make changes to this bug.