Bug 1409727

Summary: [ganesha + EC]posix compliance rename tests failed on EC volume with nfs-ganesha mount.
Product: [Community] GlusterFS Reporter: Pranith Kumar K <pkarampu>
Component: distributeAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: mainlineCC: aloganat, amukherj, aspandey, bugs, dang, ffilz, jthottan, mbenjamin, pgurusid, pkarampu, rcyriac, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.10.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1408413
: 1412913 1413061 (view as bug list) Environment:
Last Closed: 2017-03-06 17:41:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1408413, 1412913, 1413061    

Comment 1 Niels de Vos 2017-01-03 12:30:41 UTC
Please provide a public description of the problem.

Comment 2 Worker Ant 2017-01-03 16:33:58 UTC
REVIEW: http://review.gluster.org/16317 (cluster/dht: Do rename cleanup as root) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 3 Pranith Kumar K 2017-01-03 16:36:21 UTC
Steps to Reproduce:
1. Create a nfs-ganesha cluster, create 2*(4+2) EC volume, and enable ganesha on it.
2. Set mdcache options.
3. Mount the volume on client.
4. Run posix_compliance test suite
   /opt/qa/tools/system_light/run.sh -w /mnt/test_nfs -t posix_compliance -l /var/tmp/posix.log

Actual results:
posix compliance rename test fails.

Expected results:
All tests in posix compliance should pass

Additional info:

ec volume: v4 mount
---------------------------

Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  77
/opt/qa/tools/posix-testsuite/tests/open/07.t    (Wstat: 0 Tests: 23 Failed: 3)
  Failed tests:  5, 7, 9
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 6)
  Failed tests:  70-71, 74-75, 78-79
/opt/qa/tools/posix-testsuite/tests/rename/20.t  (Wstat: 0 Tests: 16 Failed: 1)
  Failed test:  16
Files=185, Tests=1962, 146 wallclock secs ( 1.66 usr  0.60 sys + 16.38 cusr 34.16 csys = 52.80 CPU)
Result: FAIL

Comment 4 Pranith Kumar K 2017-01-03 16:37:27 UTC
Works fine with this fix:
[root@localhost mnt]# prove -rfv /root/posix-testsuite/tests/rename/00.t.1 
/root/posix-testsuite/tests/rename/00.t.1 .. 
1..79
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
ok 21
ok 22
ok 23
ok 24
ok 25
ok 26
ok 27
ok 28
ok 29
ok 30
ok 31
ok 32
ok 33
ok 34
ok 35
ok 36
ok 37
ok 38
ok 39
ok 40
ok 41
ok 42
ok 43
ok 44
ok 45
ok 46
ok 47
ok 48
ok 49
ok 50
ok 51
ok 52
ok 53
ok 54
ok 55
ok 56
ok 57
ok 58
ok 59
ok 60
ok 61
ok 62
ok 63
ok 64
i am here
ok 65
ok 66
ok 67
ok 68
ok 69
ok 70
ok 71
ok 72
ok 73
ok 74
ok 75
ok 76
ok 77
ok 78
ok 79
ok
All tests successful.
Files=1, Tests=79,  9 wallclock secs ( 0.04 usr  0.02 sys +  0.09 cusr  0.31 csys =  0.46 CPU)
Result: PASS


[root@localhost mnt]# diff /root/posix-testsuite/tests/rename/00.t.1 /root/posix-testsuite/tests/rename/00.t
11,14c11,14
< n0=fstest_7835b681c64433c0ed325a7ff08a04de
< n1=fstest_45818133e626fab262bde4b293d78b1e
< n2=fstest_c34f93651b14cd35c6f9c86fe8abf266
< n3=fstest_0df806e466457e7b5d77d77a3f303ff3
---
> n0=`namegen`
> n1=`namegen`
> n2=`namegen`
> n3=`namegen`

Comment 5 Worker Ant 2017-01-05 14:07:22 UTC
COMMIT: http://review.gluster.org/16317 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 0b9af18eade6583299b88b4e685b02114db7834a
Author: Pranith Kumar K <pkarampu>
Date:   Tue Jan 3 12:50:54 2017 +0530

    cluster/dht: Do rename cleanup as root
    
    Problem:
    Rename linkfile cleanup is done as non-root which may not have priviliges to do
    the rename so it fails with EACCESS. MKDIR on that name in future will start to
    hole on this subvolume. It is not easy to hit on fuse mounts because vfs takes
    care of the permission checks even before rename fop is wound. But with
    nfs-ganesha mounts it happens.
    
    Fix:
    Do rename cleanup as root
    
    BUG: 1409727
    Change-Id: I414c1eb6dce76b4516a6c940557b249e6c3f22f4
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/16317
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>
    Reviewed-by: N Balachandran <nbalacha>
    NetBSD-regression: NetBSD Build System <jenkins.org>

Comment 6 Shyamsundar 2017-03-06 17:41:22 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/