Bug 791089

Description of problem:
Same setup as bug https://bugzilla.redhat.com/show_bug.cgi?id=791087. (791087). On nfs client rm -rf of the untarred kernel directory (which was happening parallely from the fuse client) was happening and find /nfs | xargs stat was happening. nfs server crashed on the peer where the brick crash of the bug 791087 happened.

This is the backtrace.

Core was generated by `/usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/'.
Program terminated with signal 6, Aborted.
#0  0x000000390f432905 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6_1.3.x86_64 libgcc-4.4.5-6.el6.x86_64
(gdb) #0  0x000000390f432905 in raise () from /lib64/libc.so.6
#1  0x000000390f4340e5 in abort () from /lib64/libc.so.6
#2  0x000000390f42b9be in __assert_fail_base () from /lib64/libc.so.6
#3  0x000000390f42ba80 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fb05ac8de0a in client3_1_opendir (frame=0x7fb05d438ec4, this=0xc4e220, data=0x7fff2d75a150)
    at ../../../../../xlators/protocol/client/src/client3_1-fops.c:3693
#5  0x00007fb05ac75ead in client_opendir (frame=0x7fb05d438ec4, this=0xc4e220, loc=0x7fb05809f958, fd=0x7fb0582f704c)
    at ../../../../../xlators/protocol/client/src/client.c:953
#6  0x00007fb05a9f2de0 in afr_opendir (frame=0x7fb05d42d804, this=0xc52590, loc=0x7fb05809f958, fd=0x7fb0582f704c)
    at ../../../../../xlators/cluster/afr/src/afr-dir-read.c:335
#7  0x00007fb05a7c596e in dht_rmdir (frame=0x7fb05d43cff0, this=0xc54470, loc=0x7fb05809f958, flags=0)
    at ../../../../../xlators/cluster/dht/src/dht-common.c:4163
#8  0x00007fb05e801eb1 in default_rmdir (frame=0x7fb05d439ee4, this=0xc55770, loc=0x7fb05809f958, flags=0)
    at ../../../libglusterfs/src/defaults.c:969
#9  0x00007fb05e801eb1 in default_rmdir (frame=0x7fb05d4380a8, this=0xc56be0, loc=0x7fb05809f958, flags=0)
    at ../../../libglusterfs/src/defaults.c:969
#10 0x00007fb05e801eb1 in default_rmdir (frame=0x7fb05d43aafc, this=0xc57ea0, loc=0x7fb05809f958, flags=0)
    at ../../../libglusterfs/src/defaults.c:969
#11 0x00007fb059f419f5 in io_stats_rmdir (frame=0x7fb05d43982c, this=0xc590f0, loc=0x7fb05809f958, flags=0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1940
#12 0x00007fb059cf4b62 in nfs_fop_rmdir (nfsx=0xc5a4e0, xl=0xc590f0, nfu=0x7fff2d75a8c0, pathloc=0x7fb05809f958, 
    cbk=0x7fb059cf8a7a <nfs_inode_rmdir_cbk>, local=0x7fb059cbb04c) at ../../../../../xlators/nfs/server/src/nfs-fops.c:999
#13 0x00007fb059cf8ccb in nfs_inode_rmdir (nfsx=0xc5a4e0, xl=0xc590f0, nfu=0x7fff2d75a8c0, pathloc=0x7fb05809f958, 
    cbk=0x7fb059d0ce32 <nfs3svc_rmdir_cbk>, local=0x7fb05809f520) at ../../../../../xlators/nfs/server/src/nfs-inodes.c:446
#14 0x00007fb059cf9d2e in nfs_rmdir (nfsx=0xc5a4e0, xl=0xc590f0, nfu=0x7fff2d75a8c0, path=0x7fb05809f958, 
    cbk=0x7fb059d0ce32 <nfs3svc_rmdir_cbk>, local=0x7fb05809f520) at ../../../../../xlators/nfs/server/src/nfs-generics.c:247
#15 0x00007fb059d0d110 in nfs3_rmdir_resume (carg=0x7fb05809f520) at ../../../../../xlators/nfs/server/src/nfs3.c:3535
#16 0x00007fb059d1ca91 in nfs3_fh_resolve_entry_lookup_cbk (frame=0x7fb05c405b20, cookie=0xc590f0, this=0xc5a4e0, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, buf=0xaab0fd8, xattr=0x0, postparent=0xaab1048) at ../../../../../xlators/nfs/server/src/nfs3-helpers.c:3633
#17 0x00007fb059cef6c7 in nfs_fop_lookup_cbk (frame=0x7fb05c405b20, cookie=0xc590f0, this=0xc5a4e0, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, buf=0xaab0fd8, xattr=0x0, postparent=0xaab1048) at ../../../../../xlators/nfs/server/src/nfs-fops.c:317
#18 0x00007fb059f3bc4a in io_stats_lookup_cbk (frame=0x7fb05d439378, cookie=0x7fb05d43b918, this=0xc590f0, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, buf=0xaab0fd8, xattr=0x0, postparent=0xaab1048) at ../../../../../xlators/debug/io-stats/src/io-stats.c:1468
#19 0x00007fb05a151a7e in qr_lookup_cbk (frame=0x7fb05d43b918, cookie=0x7fb05d438c14, this=0xc57ea0, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, buf=0xaab0fd8, dict=0x0, postparent=0xaab1048)
    at ../../../../../xlators/performance/quick-read/src/quick-read.c:417
#20 0x00007fb05e7f5aee in default_lookup_cbk (frame=0x7fb05d438c14, cookie=0x7fb05d43901c, this=0xc56be0, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, buf=0xaab0fd8, dict=0x0, postparent=0xaab1048) at ../../../libglusterfs/src/defaults.c:46
#21 0x00007fb05a57a30b in quota_lookup_cbk (frame=0x7fb05d43901c, cookie=0x7fb05d43a648, this=0xc55770, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, buf=0xaab0fd8, dict=0x0, postparent=0xaab1048) at ../../../../../xlators/features/quota/src/quota.c:667
#22 0x00007fb05a7aff12 in dht_lookup_cbk (frame=0x7fb05d43a648, cookie=0x7fb05d438a10, this=0xc54470, op_ret=0, op_errno=0, 
    inode=0x7fb05974bf74, stbuf=0xaab0fd8, xattr=0x0, postparent=0xaab1048) at ../../../../../xlators/cluster/dht/src/dht-common.c:1266
#23 0x00007fb05aa46323 in afr_lookup_done (frame=0x7fb05d438a10, this=0xc53810) at ../../../../../xlators/cluster/afr/src/afr-common.c:1743
#24 0x00007fb05aa46a1e in afr_lookup_cbk (frame=0x7fb05d438a10, cookie=0x1, this=0xc53810, op_ret=0, op_errno=0, inode=0x7fb05974bf74,
buf=0x7fff2d75b070, xattr=0x9d427f0, postparent=0x7fff2d75b000) at ../../../../../xlators/cluster/afr/src/afr-common.c:1906
#25 0x00007fb05ac8709c in client3_1_lookup_cbk (req=0x7fb056de9350, iov=0x7fb056de9390, count=1, myframe=0x7fb05d439424)
    at ../../../../../xlators/protocol/client/src/client3_1-fops.c:2182
#26 0x00007fb05e5cf6a4 in rpc_clnt_handle_reply (clnt=0xc68a10, pollin=0x996ade0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:790
#27 0x00007fb05e5cfa2b in rpc_clnt_notify (trans=0xc68d40, mydata=0xc68a40, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x996ade0)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:909
#28 0x00007fb05e5cbc08 in rpc_transport_notify (this=0xc68d40, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x996ade0)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:498
#29 0x00007fb0580ec23d in socket_event_poll_in (this=0xc68d40) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1675
#30 0x00007fb0580ec7c1 in socket_event_handler (fd=13, idx=4, data=0xc68d40, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1790
#31 0x00007fb05e824c20 in event_dispatch_epoll_handler (event_pool=0xc48b80, events=0xc6e890, i=0) at ../../../libglusterfs/src/event.c:794
#32 0x00007fb05e824e43 in event_dispatch_epoll (event_pool=0xc48b80) at ../../../libglusterfs/src/event.c:856
#33 0x00007fb05e8251ce in event_dispatch (event_pool=0xc48b80) at ../../../libglusterfs/src/event.c:956
#34 0x0000000000407c2e in main (argc=7, argv=0x7fff2d75b768) at ../../../glusterfsd/src/glusterfsd.c:1601
(gdb)  f 4
#4  0x00007fb05ac8de0a in client3_1_opendir (frame=0x7fb05d438ec4, this=0xc4e220, data=0x7fff2d75a150)
    at ../../../../../xlators/protocol/client/src/client3_1-fops.c:3693
3693            GF_ASSERT_AND_GOTO_WITH_ERROR (this->name,
(gdb) l
3688            if (!uuid_is_null (args->loc->inode->gfid))
3689                    memcpy (req.gfid,  args->loc->inode->gfid, 16);
3690            else
3691                    memcpy (req.gfid, args->loc->gfid, 16);
3692
3693            GF_ASSERT_AND_GOTO_WITH_ERROR (this->name,
3694                                           !uuid_is_null (*((uuid_t*)req.gfid)),
3695                                           unwind, op_errno, EINVAL);
3696
3697            conf = this->private;
(gdb) p req.gfid
$1 = '\000' <repeats 15 times>
(gdb) p args->loc->gfid
$2 = '\000' <repeats 15 times>
(gdb) p args->loc->inode->gfid
$3 = '\000' <repeats 15 times>
(gdb) p *args->loc->inode
$4 = {table = 0xc5d2f0, gfid = '\000' <repeats 15 times>, lock = 1, nlookup = 0, ref = 11, ia_type = IA_INVAL, fd_list = {
    next = 0x7fb05974bfa4, prev = 0x7fb05974bfa4}, dentry_list = {next = 0x7fb05974bfb4, prev = 0x7fb05974bfb4}, hash = {
    next = 0x7fb05974bfc4, prev = 0x7fb05974bfc4}, list = {next = 0x7fb05974bb34, prev = 0xc5d350}, _ctx = 0x554fdd0}
(gdb)p *args->loc
$5 = {path = 0x55bb280 "<gfid:9082bd91-cf75-46e9-a753-08f403b380ed>/net", name = 0x55bb2ac "net", inode = 0x7fb05974bf74, 
  parent = 0x7fb05974bad4, gfid = '\000' <repeats 15 times>, 
  pargfid = "\220\202\275\221\317uF\351\247S\b\364\003\263\200", <incomplete sequence \355>}
(gdb) 
 




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:

nfs server crashed due to gfid of loc being null

Expected results:

gfid of loc should not be null which can assert the process

Additional info:

[2012-02-15 08:50:48.951147] W [nfs3.c:3507:nfs3svc_rmdir_cbk] 0-nfs: aa2abf84: /playground/linux-2.6.31.1/drivers/media/dvb => -1 (Directory 
not empty)
[2012-02-15 08:50:48.953750] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:b06d5707-c717-4c74-86ef-cdf
0ab124fd7)
[2012-02-15 08:50:48.956340] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:b06d5707-c717-4c74-86ef-cdf
0ab124fd7)
[2012-02-15 08:50:48.957806] W [nfs3.c:3507:nfs3svc_rmdir_cbk] 0-nfs: b02abf84: /playground/linux-2.6.31.1/drivers/staging => -1 (Directory no
t empty)
[2012-02-15 08:50:48.963397] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:9082bd91-cf75-46e9-a753-08f
403b380ed)
[2012-02-15 08:50:48.965540] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:9082bd91-cf75-46e9-a753-08f
403b380ed)
[2012-02-15 08:50:48.969095] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:26edfc89-b48b-4754-8f7e-2d8
46be4e1e8)
[2012-02-15 08:50:48.972115] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:26edfc89-b48b-4754-8f7e-2d8
46be4e1e8)
[2012-02-15 08:50:48.973589] W [dict.c:461:dict_ref] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(+0x6210e) [0x7fb05aa46
10e] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_lookup_done_success_action+0x11b) [0x7fb05aa45e52] (-->/usr/local/
lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_lookup_build_response_params+0x140) [0x7fb05aa43d37]))) 0-dict: dict is NULL
[2012-02-15 08:50:48.973614] W [dht-common.c:247:dht_discover_cbk] 0-mirror-dht: <gfid:00000000-0000-0000-0000-000000000000>: gfid different o
n mirror-replicate-1
[2012-02-15 08:50:50.159765] W [dict.c:461:dict_ref] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_lookup_cbk+0xed) [
0x7fb05aa46a1e] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(+0x62323) [0x7fb05aa46323] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_discover_cbk+0x549) [0x7fb05a7a98d2]))) 0-dict: dict is NULL
[2012-02-15 08:50:50.304776] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:b9794f89-c8dc-4ec7-95a4-4d7d93baa3e8)
[2012-02-15 08:50:50.410931] W [quota.c:2147:quota_fstat_cbk] 0-mirror-quota: quota context not set in inode (gfid:b9794f89-c8dc-4ec7-95a4-4d7d93baa3e8)
[2012-02-15 08:50:50.483337] W [quota.c:780:quota_update_size] 0-mirror-quota: cannot find parent for inode (gfid:26edfc89-b48b-4754-8f7e-2d846be4e1e8), hence aborting size updation of parents
[2012-02-15 08:50:50.486663] W [quota.c:780:quota_update_size] 0-mirror-quota: cannot find parent for inode (gfid:26edfc89-b48b-4754-8f7e-2d846be4e1e8), hence aborting size updation of parents
[2012-02-15 08:50:50.492701] W [quota.c:780:quota_update_size] 0-mirror-quota: cannot find parent for inode (gfid:26edfc89-b48b-4754-8f7e-2d846be4e1e8), hence aborting size updation of parents
[2012-02-15 08:50:50.497576] W [dict.c:461:dict_ref] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(+0x6210e) [0x7fb05aa4610e] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_lookup_done_success_action+0x11b) [0x7fb05aa45e52] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_lookup_build_response_params+0x140) [0x7fb05aa43d37]))) 0-dict: dict is NULL
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 2012-02-15 08:50:50
configuration details:
argp 1