Bug 971805

Summary:	nfs: "rm -rf" throws "E [client3_1-fops.c:5214:client3_1_inodelk]" Assertion failed
Product:	[Community] GlusterFS	Reporter:	Pranith Kumar K <pkarampu>
Component:	distribute	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	mainline	CC:	gluster-bugs, rhs-bugs, saujain, spradhan, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	968289	Environment:
Last Closed:	2013-07-24 17:33:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	968289
Bug Blocks:	1028582

Description Pranith Kumar K 2013-06-07 10:13:48 UTC

+++ This bug was initially created as a clone of Bug #968289 +++

Description of problem:
volume type 6x2

just trying to do "rm -rf *" on nfs mount

How reproducible:
1/1

Steps to Reproduce:
1. create volume of type 6x3, start the volume
2. add some data to it
3. execute "rm -rf *" on the mount point

Actual results:
rm -rf finished, but throws error in nfs.log

nfs.log says,
[2013-05-29 09:25:32.172816] I [glusterfsd-mgmt.c:1568:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2013-05-29 09:25:32.173503] I [glusterfsd-mgmt.c:1568:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2013-05-29 09:27:33.502273] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_rec+0x80) [0x7f150ff028e0] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_nonblocking_inodelk+0x608) [0x7f150ff1e5b8] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.502847] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_rec+0x80) [0x7f150ff028e0] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_nonblocking_inodelk+0x608) [0x7f150ff1e5b8] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.502963] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_blocking_lock+0x84) [0x7f150ff1f354] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_blocking+0xad7) [0x7f150ff1f207] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.503050] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(+0x4545e) [0x7f150ff1f45e] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_blocking+0xad7) [0x7f150ff1f207] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.503089] I [afr-lk-common.c:996:afr_lock_blocking] 0-dist-rep-replicate-1: unable to lock on even one child
[2013-05-29 09:27:33.503119] I [afr-transaction.c:1031:afr_post_blocking_inodelk_cbk] 0-dist-rep-replicate-1: Blocking inodelks failed.
[2013-05-29 09:27:33.503161] E [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-dist-rep-dht: setattr of uid/gid on <gfid:64c844f0-1ddf-48cf-b195-330a781a2a07>/c4 :<gfid:00000000-0000-0000-0000-000000000000> failed (Invalid argument)

--- Additional comment from Pranith Kumar K on 2013-06-07 06:10:05 EDT ---

Amar,
    Following log suggests that for linkfile setattr gfid is not present either in loc->gfid/loc->inode->gfid.

[2013-05-29 09:27:33.503161] E [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-dist-rep-dht: setattr of uid/gid on <gfid:64c844f0-1ddf-48cf-b195-330a781a2a07>/c4 :<gfid:00000000-0000-0000-0000-000000000000> failed (Invalid argument)

I could re-create the issue on master by running bug-884597.t

Here are the details:
(gdb) fr 7
#7  0x00007f0b4093561a in dht_lookup_linkfile_create_cbk (frame=0x7f0b43c523a8, 

(gdb) p local->loc
$1 = {path = 0x7f0b34002220 "/2", name = 0x7f0b34002221 "2", inode = 0x7f0b3b0ab0e8, 
  parent = 0x7f0b3b0ab04c, gfid = '\000' <repeats 15 times>, 
  pargfid = '\000' <repeats 15 times>, "\001"}
(gdb) p local->loc->inode->gfid
$2 = '\000' <repeats 15 times>

The following patch fixes the problem. Will send the patch.

pranithk@pranithk-laptop - ~/workspace/gerrit-repo/tests/bugs (master)
15:28:59 :) ⚡ git diff
diff --git a/xlators/cluster/dht/src/dht-linkfile.c b/xlators/cluster/dht/src/dht-linkfile.c
index 39d72ae..ae5bd49 100644
--- a/xlators/cluster/dht/src/dht-linkfile.c
+++ b/xlators/cluster/dht/src/dht-linkfile.c
@@ -302,6 +302,8 @@ dht_linkfile_attr_heal (call_frame_t *frame, xlator_t *this)
              is_equal (frame->root->gid, local->stbuf.ia_gid)))
                 return 0;
 
+        uuid_copy (local->loc.gfid, local->stbuf.ia_gfid);
+
         copy = copy_frame (frame);
 
         if (!copy)

Comment 1 Anand Avati 2013-06-07 10:25:33 UTC

REVIEW: http://review.gluster.org/5178 (cluster/dht: Make sure loc has gfid) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 2 Anand Avati 2013-06-11 01:46:21 UTC

COMMIT: http://review.gluster.org/5178 committed in master by Vijay Bellur (vbellur) 
------
commit 22d821969a7204d09339b1a52a1d38e609e5aa86
Author: Pranith Kumar K <pkarampu>
Date:   Fri Jun 7 15:45:26 2013 +0530

    cluster/dht: Make sure loc has gfid
    
    Problem:
    In some code paths neither loc->gfid nor loc->inode->gfid
    is populated which leads to EINVAL for linkfile setattr
    in dht_linkfile_attr_heal.
    
    Fix:
    Populate loc->gfid before dht_linkfile_attr_heal.
    
    Change-Id: I062770e6f6eaead304eff1dae81f8588a3b97eed
    BUG: 971805
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/5178
    Reviewed-by: Shishir Gowda <sgowda>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 3 Anand Avati 2014-01-13 10:52:14 UTC

REVIEW: http://review.gluster.org/6691 (cluster/dht: Make sure loc has gfid) posted (#1) for review on release-3.4 by Pranith Kumar Karampuri (pkarampu)

Comment 4 Anand Avati 2014-01-13 12:58:55 UTC

COMMIT: http://review.gluster.org/6691 committed in release-3.4 by Vijay Bellur (vbellur) 
------
commit 8eda793ed6762ca71521698eb80e2262b5b63092
Author: Pranith Kumar K <pkarampu>
Date:   Mon Jan 13 16:13:30 2014 +0530

    cluster/dht: Make sure loc has gfid
    
    Problem:
    In some code paths neither loc->gfid nor loc->inode->gfid
    is populated which leads to EINVAL for linkfile setattr
    in dht_linkfile_attr_heal.
    
    Fix:
    Populate loc->gfid before dht_linkfile_attr_heal.
    
    BUG: 971805
    Change-Id: I8e4b7510ee5c38aa9ccf5283c7165c7df25ec62b
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/6691
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>