Bug 1391808

(In reply to Prasad Desala from comment #0)
> Also, after issuing 'ls' on the parent directory created in 4a, we are
> seeing the below info message in FUSE logs,
> 
> [2016-11-03 10:30:54.284974] I [MSGID: 109063]
> [dht-layout.c:713:dht_layout_normalize] 2-distrep-dht: Found anomalies in
> /dir/dir (gfid = ef02bc07-ba92-4e40-bd81-1661b0d41582). Holes=1 overlaps=0
> 
> But, I did not see any holes on the directory created in 4d,
> 
> [root@dhcp42-7 ~]# getfattr -d -e hex -m trusted.glusterfs.dht
> /bricks/brick*/*/dir/dir

This is because the directory was created without the DHT xattrs and as the client detected an anomaly in the directory, it healed it as root, and so the layout *finally* on disk was not empty.

Before doing the ls on ./dir/ if you would take a look at the xattrs on the brick, you would find them missing.

I was able to replicate the problem based on the steps given (thanks).

What occurs here is as follows,
- DHT sends the mkdir call to the bricks
- Post the mkdir, it sends the setxattr call to set the layout xattrs for these directories

The failure is noted in the bricks and on the FUSE client logs, as the setxattr fails.

The setxattr fails, due to the permissions on the directory being 0464 (set during the mkdir as requested by the call), where user has only READ permissions, and posix_acl xlator checks that the UID of the setxattr caller (internal DHT request above) is the same as the UID of the directory inode, and hence evaluates permissions based on this, which evaluates to READ ONLY and denies the setxattr from proceeding.

Code reference:
  - https://github.com/gluster/glusterfs/blob/master/xlators/system/posix-acl/src/posix-acl.c#L2088 posix_acl_setxattr -> calls setxattr_scrutiny
  - https://github.com/gluster/glusterfs/blob/master/xlators/system/posix-acl/src/posix-acl.c#L1920 setxattr_scrutiny -> acl_permits calls
  - acl_permits: Has the POSIX_ACL_USER_OBJ in its list first, and checks the UID equality of the caller to the inode (which matches) and hence uses permissions of the user to proceed doing a perm_check

The downside of all this, is that the DHT layout on this directory is not set, only on a subsequent layout anomaly detection by DHT is this corrected, as anomaly correction is an internal operation that uses UID/GID as root/root and the ACL xlator passes the request without the checks.

General discussion:
The manner in which gluster ACL xlator checks the permissions (user first and then group etc.) is correct as per standards, the reference would be, https://www.gnu.org/software/libc/manual/html_node/Access-Permission.html#Access-Permission

Further on checking the kernel code, the same rules apply there as well, and a directory created with READ for user, subsequently fails with EPERM for any setfattr calls. Kernel code reference: http://lxr.free-electrons.com/source/fs/posix_acl.c#L346

So basically, the implementation in out ACL xlator seems to be right. I.e just because group bits provide WRITE permissions, it is not allowed as the UID of the directory to perform any WRITE operations.

What this means:
So currently other than the logs, there is no real bug to start with. But, the finding is interesting, and we could make the setxattr of DHT xattrs post mkdir an internal operation to be done as root/root (just like healing an anomaly). That would fix the logging problem as well. Is it a security violation is something that is needed to be considered.

For example xattrop that is driven by the client (and is actually a setxattr call in the end) has no ACL xlator checks against that FOP (and as a result does not suffer this problem).

Regression notes:
This should not be a regression, and should be present in the prior releases as well, just a heads up!

Upstream patch:

http://review.gluster.org/15794

Verified this BZ on glusterfs version 3.8.4-8.el7rhgs.x86_64. 
Followed the above mentioned steps and we are not seeing any [setxattr_cbk] "Permission denied" warning messages in logs.

Moving this BZ to Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html