Bug 1543487

Summary: dht_lookup_unlink_of_false_linkto_cbk fails with "Permission denied"
Product: [Community] GlusterFS Reporter: Nithya Balachandran <nbalacha>
Component: distributeAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.0CC: amukherj, rhinduja, rhs-bugs, storage-qa-internal, tdesala
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-4.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1542318 Environment:
Last Closed: 2018-03-15 11:26:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1540664, 1542318    
Bug Blocks: 1543016    

Description Nithya Balachandran 2018-02-08 14:45:23 UTC
+++ This bug was initially created as a clone of Bug #1542318 +++

+++ This bug was initially created as a clone of Bug #1540664 +++

Description of problem:

[2018-01-30 19:41:53.630949] I [MSGID: 109045] [dht-common.c:2391:dht_lookup_cbk] 0-vol1-dht: linkfile not having link subvol for /data/file1
[2018-01-30 19:41:53.672524] W [MSGID: 114031] [client-rpc-fops.c:630:client3_3_unlink_cbk] 0-vol1-client-6: remote operation failed [Permission denied]
[2018-01-30 19:41:53.672638] W [MSGID: 114031] [client-rpc-fops.c:630:client3_3_unlink_cbk] 0-vol1-client-7: remote operation failed [Permission denied]
[2018-01-30 19:41:53.672632] W [MSGID: 114031] [client-rpc-fops.c:630:client3_3_unlink_cbk] 0-vol1-client-8: remote operation failed [Permission denied]
[2018-01-30 19:41:53.672828] W [MSGID: 114031] [client-rpc-fops.c:630:client3_3_unlink_cbk] 0-vol1-client-11: remote operation failed [Permission denied]
[2018-01-30 19:41:53.672839] I [MSGID: 109069] [dht-common.c:1353:dht_lookup_unlink_of_false_linkto_cbk] 0-vol1-dht: lookup_unlink returned with op_ret -> -1 and op-errno -> 13 for /data/file1
[2018-01-30 19:41:53.672868] W [MSGID: 112199] [nfs3-helpers.c:3515:nfs3_log_newfh_res] 0-nfs-nfsv3: /data/file1 => (XID: 52dbaffa, LOOKUP: NFS: 5(I/O error), POSIX: 5(Input/output error)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000, mountid 00000000-0000-0000-0000-000000000000 [Resource temporarily unavailable]

Actual results:

Bricks are unable to fully heal.

Expected results:

Normal operation.

Additional info:

--- Additional comment from Nithya Balachandran on 2018-02-06 04:48:10 EST ---

Unprivileged users do not have permissions to delete stale linkto files. Failure to unlink the file causes DHT to return EIO.


Steps to reproduce this issue:
=============================

1. gluster volume create vol1 server1:/bricks/brick1/vol1-{1,2}
2. gluster volume start vol1                                                      
3. mount -t glusterfs -s server1:/vol1 /mnt/fuse1                                     
                                                                                
# create a file for testing                                                     
4. dd if=/dev/urandom of=/mnt/fuse1/FILE-1 count=1 bs=16k                            
5. gluster volume set vol1 performance.parallel-readdir on

#Rename the file in order to create a linkto file                        
6. mv vol1/FILE-1 vol1/FILE-2    
                                               
#Force the linkto xattr values to become stale by changing the dht subvols in the graph                               
7. gluster volume set vol1 performance.parallel-readdir off 

Fuse mount as a non-privileged user
                    
8. echo user_allow_other >> /etc/fuse.conf                                         
9. gluster volume set vol1 allow-insecure on  

Create a user user1
10. useradd user1     
11: su user1                  
12. cd /home/user1/
13. mkdir mymount

#Mount the volume as user1
14: glusterfs --log-level=INFO --log-file=my_user_accessable_logfile.log --volfile-id=vol1 --volfile-server=server1 ~/mymount

15: cd mymount/
16. ls FILE-2

Results:
=========
ls: cannot access FILE-2: Input/output error

--- Additional comment from Worker Ant on 2018-02-06 04:53:36 EST ---

REVIEW: https://review.gluster.org/19508 (cluster/dht: Unlink linkto files as root) posted (#1) for review on master by N Balachandran

--- Additional comment from Nithya Balachandran on 2018-02-06 08:32:53 EST ---

See Joe Julian's blog for details on how to FUSE mount the volume as a non-privileged user.

https://joejulian.name/post/mounting-a-glusterfs-volume-as-an-unprivileged-user/

--- Additional comment from Worker Ant on 2018-02-07 09:33:32 EST ---

COMMIT: https://review.gluster.org/19508 committed in master by "N Balachandran" <nbalacha> with a commit message- cluster/dht: Unlink linkto files as root

Non-privileged users cannot delete linkto
files. However the failure to unlink a stale linkto
causes DHT to fail the lookup with EIO and hence
prevent access to the file.

Change-Id: Id295362d41e52263790694602f36f1219f0646a2
BUG: 1542318
Signed-off-by: N Balachandran <nbalacha>

--- Additional comment from Worker Ant on 2018-02-07 10:30:14 EST ---

REVIEW: https://review.gluster.org/19524 (tests/dht: Non-root can delete stale linkto files) posted (#1) for review on master by N Balachandran

--- Additional comment from Worker Ant on 2018-02-08 03:19:36 EST ---

COMMIT: https://review.gluster.org/19524 committed in master by "N Balachandran" <nbalacha> with a commit message- tests/dht: Non-root can delete stale linkto files

Test to check that non-root users can delete stale
linkto files

Change-Id: Ic9bc76bc485cab839927af60cfce78a058eee2e4
BUG: 1542318
Signed-off-by: N Balachandran <nbalacha>

Comment 1 Worker Ant 2018-02-08 14:56:53 UTC
REVIEW: https://review.gluster.org/19528 (cluster/dht: Unlink linkto files as root) posted (#1) for review on release-4.0 by N Balachandran

Comment 2 Worker Ant 2018-02-09 13:44:33 UTC
COMMIT: https://review.gluster.org/19528 committed in release-4.0 by "Shyamsundar Ranganathan" <srangana> with a commit message- cluster/dht: Unlink linkto files as root

Non-privileged users cannot delete linkto
files. However the failure to unlink a stale linkto
causes DHT to fail the lookup with EIO and hence
prevent access to the file.

> Change-Id: Id295362d41e52263790694602f36f1219f0646a2
> BUG: 1542318
> Signed-off-by: N Balachandran <nbalacha>

Change-Id: Id295362d41e52263790694602f36f1219f0646a2
BUG: 1543487
Signed-off-by: N Balachandran <nbalacha>

Comment 3 Worker Ant 2018-02-12 06:29:02 UTC
REVIEW: https://review.gluster.org/19542 (tests/dht: Non-root can delete stale linkto files) posted (#1) for review on release-4.0 by N Balachandran

Comment 4 Worker Ant 2018-02-15 14:08:47 UTC
COMMIT: https://review.gluster.org/19542 committed in release-4.0 by "N Balachandran" <nbalacha> with a commit message- tests/dht: Non-root can delete stale linkto files

Test to check that non-root users can delete stale
linkto files

> Change-Id: Ic9bc76bc485cab839927af60cfce78a058eee2e4
> BUG: 1542318
> Signed-off-by: N Balachandran <nbalacha>

Change-Id: Ic9bc76bc485cab839927af60cfce78a058eee2e4
BUG: 1543487
Signed-off-by: N Balachandran <nbalacha>

Comment 5 Shyamsundar 2018-03-15 11:26:23 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report.

glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html
[2] https://www.gluster.org/pipermail/gluster-users/