Bug 1391808 - [setxattr_cbk] "Permission denied" warning messages are seen in logs while running pjd-fstest suite
Summary: [setxattr_cbk] "Permission denied" warning messages are seen in logs while ru...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: RHGS 3.2.0
Assignee: Nithya Balachandran
QA Contact: Prasad Desala
URL:
Whiteboard:
Depends On: 1392772 1397252
Blocks: 1351528
TreeView+ depends on / blocked
 
Reported: 2016-11-04 05:29 UTC by Prasad Desala
Modified: 2017-03-23 06:16 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.8.4-6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1392772 (view as bug list)
Environment:
Last Closed: 2017-03-23 06:16:22 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Prasad Desala 2016-11-04 05:29:33 UTC
Description of problem:
=======================
Using pjdfstest filesystem test suite when created a directory with its access and group permissions below warning messages are seen in FUSE and brick logs,

FUSE logs:
=========
[2016-11-03 10:27:10.948703] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_setxattr_cbk] 2-distrep-client-7: remote operation failed [Permission denied]
Brick logs:
===========
[2016-11-03 10:26:56.494850] I [MSGID: 115060] [server-rpc-fops.c:865:_gf_server_log_setxattr_failure] 0-distrep-server: 2505: SETXATTR /dir/dir (ef02bc07-ba92-4e40-bd81-1661b0d41582) ==> trusted.glusterfs.dht
[2016-11-03 10:26:56.495084] I [MSGID: 115060] [server-rpc-fops.c:893:server_setxattr_cbk] 0-distrep-server: Permission denied

Version-Release number of selected component (if applicable):
3.8.4-3.el7rhgs.x86_64

How reproducible:
=================
Always

Steps to Reproduce:
===================
1) Create  a distributed replica volume and start it.
2) FUSE mount the volume on a client.
3) As a root user run the below command from mount point to clone pjd-fstest
	git://git.code.sf.net/p/ntfs-3g/pjd-fstest
4) Run the below commands
   a) ./pjd-fstest/fstest mkdir dir 0777
   b) ./pjd-fstest/fstest chown dir 65534 65534
   c) ./pjd-fstest/fstest -u 65534 -g 65534 create dir/file 0464
   d) ./pjd-fstest/fstest -u 65534 -g 65534 mkdir dir/dir 0464

When executed 4d the permission denied messages are seen in FUSE and brick logs.

Also, after issuing 'ls' on the parent directory created in 4a, we are seeing the below info message in FUSE logs,

[2016-11-03 10:30:54.284974] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-distrep-dht: Found anomalies in /dir/dir (gfid = ef02bc07-ba92-4e40-bd81-1661b0d41582). Holes=1 overlaps=0

But, I did not see any holes on the directory created in 4d,

[root@dhcp42-7 ~]# getfattr -d -e hex -m trusted.glusterfs.dht /bricks/brick*/*/dir/dir
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/b0/dir/dir
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff

# file: bricks/brick1/b1/dir/dir
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553

# file: bricks/brick2/b2/dir/dir
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7

[root@dhcp43-141 ~]# getfattr -d -e hex -m trusted.glusterfs.dht /bricks/brick*/*/dir/dir
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/b0/dir/dir
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9

# file: bricks/brick1/b1/dir/dir
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd

# file: bricks/brick2/b2/dir/dir
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551


Actual results:
===============
[setxattr_cbk] "Permission denied" warning messages are seen in logs while running pjd-fstest suite

Expected results:
=================
Permission denied messages should not be seen.

Additional info:

Comment 3 Shyamsundar 2016-11-07 18:44:52 UTC
(In reply to Prasad Desala from comment #0)
> Also, after issuing 'ls' on the parent directory created in 4a, we are
> seeing the below info message in FUSE logs,
> 
> [2016-11-03 10:30:54.284974] I [MSGID: 109063]
> [dht-layout.c:713:dht_layout_normalize] 2-distrep-dht: Found anomalies in
> /dir/dir (gfid = ef02bc07-ba92-4e40-bd81-1661b0d41582). Holes=1 overlaps=0
> 
> But, I did not see any holes on the directory created in 4d,
> 
> [root@dhcp42-7 ~]# getfattr -d -e hex -m trusted.glusterfs.dht
> /bricks/brick*/*/dir/dir

This is because the directory was created without the DHT xattrs and as the client detected an anomaly in the directory, it healed it as root, and so the layout *finally* on disk was not empty.

Before doing the ls on ./dir/ if you would take a look at the xattrs on the brick, you would find them missing.

Comment 4 Shyamsundar 2016-11-07 19:07:13 UTC
I was able to replicate the problem based on the steps given (thanks).

What occurs here is as follows,
- DHT sends the mkdir call to the bricks
- Post the mkdir, it sends the setxattr call to set the layout xattrs for these directories

The failure is noted in the bricks and on the FUSE client logs, as the setxattr fails.

The setxattr fails, due to the permissions on the directory being 0464 (set during the mkdir as requested by the call), where user has only READ permissions, and posix_acl xlator checks that the UID of the setxattr caller (internal DHT request above) is the same as the UID of the directory inode, and hence evaluates permissions based on this, which evaluates to READ ONLY and denies the setxattr from proceeding.

Code reference:
  - https://github.com/gluster/glusterfs/blob/master/xlators/system/posix-acl/src/posix-acl.c#L2088 posix_acl_setxattr -> calls setxattr_scrutiny
  - https://github.com/gluster/glusterfs/blob/master/xlators/system/posix-acl/src/posix-acl.c#L1920 setxattr_scrutiny -> acl_permits calls
  - acl_permits: Has the POSIX_ACL_USER_OBJ in its list first, and checks the UID equality of the caller to the inode (which matches) and hence uses permissions of the user to proceed doing a perm_check

The downside of all this, is that the DHT layout on this directory is not set, only on a subsequent layout anomaly detection by DHT is this corrected, as anomaly correction is an internal operation that uses UID/GID as root/root and the ACL xlator passes the request without the checks.

General discussion:
The manner in which gluster ACL xlator checks the permissions (user first and then group etc.) is correct as per standards, the reference would be, https://www.gnu.org/software/libc/manual/html_node/Access-Permission.html#Access-Permission

Further on checking the kernel code, the same rules apply there as well, and a directory created with READ for user, subsequently fails with EPERM for any setfattr calls. Kernel code reference: http://lxr.free-electrons.com/source/fs/posix_acl.c#L346

So basically, the implementation in out ACL xlator seems to be right. I.e just because group bits provide WRITE permissions, it is not allowed as the UID of the directory to perform any WRITE operations.

What this means:
So currently other than the logs, there is no real bug to start with. But, the finding is interesting, and we could make the setxattr of DHT xattrs post mkdir an internal operation to be done as root/root (just like healing an anomaly). That would fix the logging problem as well. Is it a security violation is something that is needed to be considered.

For example xattrop that is driven by the client (and is actually a setxattr call in the end) has no ACL xlator checks against that FOP (and as a result does not suffer this problem).

Regression notes:
This should not be a regression, and should be present in the prior releases as well, just a heads up!

Comment 5 Nithya Balachandran 2016-11-08 09:01:47 UTC
Upstream patch:

http://review.gluster.org/15794

Comment 10 Prasad Desala 2016-12-12 13:06:42 UTC
Verified this BZ on glusterfs version 3.8.4-8.el7rhgs.x86_64. 
Followed the above mentioned steps and we are not seeing any [setxattr_cbk] "Permission denied" warning messages in logs.

Moving this BZ to Verified.

Comment 12 errata-xmlrpc 2017-03-23 06:16:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.