Bug 971805 - nfs: "rm -rf" throws "E [client3_1-fops.c:5214:client3_1_inodelk]" Assertion failed
nfs: "rm -rf" throws "E [client3_1-fops.c:5214:client3_1_inodelk]" Assertio...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
mainline
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Pranith Kumar K
:
Depends On: 968289
Blocks: 1028582
  Show dependency treegraph
 
Reported: 2013-06-07 06:13 EDT by Pranith Kumar K
Modified: 2014-01-13 07:58 EST (History)
5 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 968289
Environment:
Last Closed: 2013-07-24 13:33:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Pranith Kumar K 2013-06-07 06:13:48 EDT
+++ This bug was initially created as a clone of Bug #968289 +++

Description of problem:
volume type 6x2

just trying to do "rm -rf *" on nfs mount

How reproducible:
1/1

Steps to Reproduce:
1. create volume of type 6x3, start the volume
2. add some data to it
3. execute "rm -rf *" on the mount point

Actual results:
rm -rf finished, but throws error in nfs.log

nfs.log says,
[2013-05-29 09:25:32.172816] I [glusterfsd-mgmt.c:1568:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2013-05-29 09:25:32.173503] I [glusterfsd-mgmt.c:1568:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2013-05-29 09:27:33.502273] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_rec+0x80) [0x7f150ff028e0] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_nonblocking_inodelk+0x608) [0x7f150ff1e5b8] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.502847] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_rec+0x80) [0x7f150ff028e0] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_nonblocking_inodelk+0x608) [0x7f150ff1e5b8] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.502963] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_blocking_lock+0x84) [0x7f150ff1f354] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_blocking+0xad7) [0x7f150ff1f207] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.503050] E [client3_1-fops.c:5214:client3_1_inodelk] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(+0x4545e) [0x7f150ff1f45e] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/cluster/replicate.so(afr_lock_blocking+0xad7) [0x7f150ff1f207] (-->/usr/lib64/glusterfs/3.3.0.8rhs/xlator/protocol/client.so(client_inodelk+0x9e) [0x7f151015279e]))) 0-: Assertion failed: 0
[2013-05-29 09:27:33.503089] I [afr-lk-common.c:996:afr_lock_blocking] 0-dist-rep-replicate-1: unable to lock on even one child
[2013-05-29 09:27:33.503119] I [afr-transaction.c:1031:afr_post_blocking_inodelk_cbk] 0-dist-rep-replicate-1: Blocking inodelks failed.
[2013-05-29 09:27:33.503161] E [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-dist-rep-dht: setattr of uid/gid on <gfid:64c844f0-1ddf-48cf-b195-330a781a2a07>/c4 :<gfid:00000000-0000-0000-0000-000000000000> failed (Invalid argument)

--- Additional comment from Pranith Kumar K on 2013-06-07 06:10:05 EDT ---

Amar,
    Following log suggests that for linkfile setattr gfid is not present either in loc->gfid/loc->inode->gfid.

[2013-05-29 09:27:33.503161] E [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-dist-rep-dht: setattr of uid/gid on <gfid:64c844f0-1ddf-48cf-b195-330a781a2a07>/c4 :<gfid:00000000-0000-0000-0000-000000000000> failed (Invalid argument)

I could re-create the issue on master by running bug-884597.t

Here are the details:
(gdb) fr 7
#7  0x00007f0b4093561a in dht_lookup_linkfile_create_cbk (frame=0x7f0b43c523a8, 

(gdb) p local->loc
$1 = {path = 0x7f0b34002220 "/2", name = 0x7f0b34002221 "2", inode = 0x7f0b3b0ab0e8, 
  parent = 0x7f0b3b0ab04c, gfid = '\000' <repeats 15 times>, 
  pargfid = '\000' <repeats 15 times>, "\001"}
(gdb) p local->loc->inode->gfid
$2 = '\000' <repeats 15 times>

The following patch fixes the problem. Will send the patch.

pranithk@pranithk-laptop - ~/workspace/gerrit-repo/tests/bugs (master)
15:28:59 :) ⚡ git diff
diff --git a/xlators/cluster/dht/src/dht-linkfile.c b/xlators/cluster/dht/src/dht-linkfile.c
index 39d72ae..ae5bd49 100644
--- a/xlators/cluster/dht/src/dht-linkfile.c
+++ b/xlators/cluster/dht/src/dht-linkfile.c
@@ -302,6 +302,8 @@ dht_linkfile_attr_heal (call_frame_t *frame, xlator_t *this)
              is_equal (frame->root->gid, local->stbuf.ia_gid)))
                 return 0;
 
+        uuid_copy (local->loc.gfid, local->stbuf.ia_gfid);
+
         copy = copy_frame (frame);
 
         if (!copy)
Comment 1 Anand Avati 2013-06-07 06:25:33 EDT
REVIEW: http://review.gluster.org/5178 (cluster/dht: Make sure loc has gfid) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 2 Anand Avati 2013-06-10 21:46:21 EDT
COMMIT: http://review.gluster.org/5178 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 22d821969a7204d09339b1a52a1d38e609e5aa86
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Fri Jun 7 15:45:26 2013 +0530

    cluster/dht: Make sure loc has gfid
    
    Problem:
    In some code paths neither loc->gfid nor loc->inode->gfid
    is populated which leads to EINVAL for linkfile setattr
    in dht_linkfile_attr_heal.
    
    Fix:
    Populate loc->gfid before dht_linkfile_attr_heal.
    
    Change-Id: I062770e6f6eaead304eff1dae81f8588a3b97eed
    BUG: 971805
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/5178
    Reviewed-by: Shishir Gowda <sgowda@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Comment 3 Anand Avati 2014-01-13 05:52:14 EST
REVIEW: http://review.gluster.org/6691 (cluster/dht: Make sure loc has gfid) posted (#1) for review on release-3.4 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 4 Anand Avati 2014-01-13 07:58:55 EST
COMMIT: http://review.gluster.org/6691 committed in release-3.4 by Vijay Bellur (vbellur@redhat.com) 
------
commit 8eda793ed6762ca71521698eb80e2262b5b63092
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Mon Jan 13 16:13:30 2014 +0530

    cluster/dht: Make sure loc has gfid
    
    Problem:
    In some code paths neither loc->gfid nor loc->inode->gfid
    is populated which leads to EINVAL for linkfile setattr
    in dht_linkfile_attr_heal.
    
    Fix:
    Populate loc->gfid before dht_linkfile_attr_heal.
    
    BUG: 971805
    Change-Id: I8e4b7510ee5c38aa9ccf5283c7165c7df25ec62b
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/6691
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Note You need to log in before you can comment on or make changes to this bug.