Bug 1405147 - glusterfs (posix-acl xlator layer) checks for "write permission" instead for "file owner" during open() when writing to a file
Summary: glusterfs (posix-acl xlator layer) checks for "write permission" instead for ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: open-behind
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Raghavendra G
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-15 17:05 UTC by omar
Modified: 2019-09-18 07:34 UTC (History)
14 users (show)

Fixed In Version: glusterfs-5.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-23 15:06:00 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
kdhananj: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1543996 0 medium CLOSED truncates read-only files on copy 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1580120 0 unspecified CLOSED [Ganesha] glusterfs (posix-acl xlator layer) checks for "write permission" instead for "file owner" during open() when ... 2021-02-22 00:41:40 UTC

Internal Links: 1543996 1580120

Description omar 2016-12-15 17:05:35 UTC
Description of problem:
When using a fuse-mounted glusterfs volume, an error is found if the owner of a file (that is not root) tries to write in its own file when it has read-only permissions.

The problem does not exist if the user is root.

Version-Release number of selected component (if applicable):
All versions tested (glusterfs-3.7.{4-16}, glusterfs-3.8.{4-5})

How reproducible:
Always

Steps to Reproduce:

1. Fuse mount a glusterfs volume to be the home of the user, cd to home dir and check that the file we will use for testing does not exist:
user@computer:~$ ls -l 444.txt
ls: cannot access 444.txt: No such file or directory
user@computer:~$

2. Execute command (error comes at echo, not before):
user@computer:~$ rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo test > 444.txt
-bash: 444.txt: Permission denied

3. Check result: File is empty.
user@computer:~$ ls -l 444.txt
-r--r--r-- 1 user user 0 dic 15  2016 444.txt
user@computer:~$ rm -f 444.txt
user@computer:~$

4. Check that the problem dissapears with root privileges:
user@computer:~$ sudo su
root@computer:/home/user# rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo test > 444.txt
root@computer:/home/user# ls -l 444.txt
-r--r--r-- 1 root root 5 dic 15  2016 444.txt
root@computer:/home/user# rm -f 444.txt
root@computer:/home/user#

Actual results:
A user who owns a file that is read-only cannot write into it.

Expected results:
A user who owns a file that is read-only should be able to write into it even if it is read-only.

Additional info:
Special thanks to Bruce Fields and other components of nfs-kernel-server team because initially we thought it was their fault. The same to Soumya Koduri and Miklos Szeredi, that helped to confirm the problem.

See full previous thread on nfs-kernel-server list with history about the issue:
http://marc.info/?t=144801803400001 (for posterity's sake, permanent link: http://marc.info/?i=862ef8b0-61ee-8b7e-a81c-fcdc5a726e37@dit.upm.es).

Comment 2 Jiffin 2016-12-16 09:25:59 UTC
Can u change the product into glusterfs and component into access-control.

IMO it is accepted behavior. Posix compliant file system(xfs) behaves in a similar way.  
[jiffin@jiffin17 ~]$ touch file
[jiffin@jiffin17 ~]$ chmod 444 file
[jiffin@jiffin17 ~]$ echo read > file 
bash: file: Permission denied

[root@jiffin17 ~]# touch file
[root@jiffin17 ~]# chmod 444  file
[root@jiffin17 ~]# echo read > file

GlusterFS being posix compliant, it should behave like this

Comment 3 Soumya Koduri 2016-12-16 10:14:36 UTC
Jiffin,

This is not for the WRITE after OPEN... but while creating a file, file_owner should get the write permissions. Try this particular case - 

#rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt 

It works fine on fuse mount but not when that fuse mount is exported via kernel-nfs and mounted using nfs client.

Comment 4 Soumya Koduri 2016-12-16 10:27:35 UTC
and agree that it should have been filed against upstream glusterfs component.

Comment 5 omar 2016-12-16 11:28:34 UTC
Sorry to all, I have to apologize because I tried to super-simplify the problem in the explanation without remembering properly previous tests made and possible issues discarded... 

The origin of this problem is about copying a read-only file, not about writing data on it.

Tell me if you think maybe is better to file a new bug or to explain it again in this bug report.

If the later, the explanation would be the following (based on http://marc.info/?l=linux-nfs&m=147932042025835):

Description of problem:
When you export a glusterfs fuse-mounted volume through a nfs-kernel-server, if the nfs-client mounting it tries to copy an owned read-only file -not being root user-, a permission denied error shows up at the client side and it is recorded at the glusterfs logs. 

If the user is root no problem exist.

Version-Release number of selected component (if applicable):
All versions tested (glusterfs-3.7.{4-16}, glusterfs-3.8.{4-5})

How reproducible:
Always

Steps to Reproduce:
The setup is:

      nfs client---NFSv3--->knfsd---FUSE-->gluster

The tests are:

# At the knfsd server:
root@cuentas03-lab:/etc# cat /proc/mounts
[...]
recipiente6hd:/home-lab-3.tcp /home-3-old fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
root@cuentas03-lab:/etc#

# At the nfs-client:
cdc@l056:~$ cat /proc/mounts | grep "/home/cdc"
cuentas03:/home-3-old/cdc /home/cdc nfs rw,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=138.4.30.18,mountvers=3,mountport=49531,mountproto=tcp,fsc,local_lock=all,addr=138.4.30.18 0 0
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 16  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 16  2016 kk.txt
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ sudo su
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
-r--r--r-- 1 root root 7 nov 16  2016 444.txt
-r--r--r-- 1 root root 7 nov 16  2016 kk.txt
root@l056:/home/cdc# exit
cdc@l056:~$ 

Actual results:
As a nfs-client of a fuse-mounted glusterfs volume, a user who owns a file that is read-only cannot copy it to a new file.

Expected results:
As a nfs-client of a fuse-mounted glusterfs volume, a user who owns a file that is read-only should be able to copy it even if it is read-only.

Hope this helps to clarify. Sorry for the noise.

Comment 6 fredrik 2017-05-01 16:38:51 UTC
We are seeing the same issue but this time on MooseFS exported as nfs via kernel nfs.

The same test works fine when accessing MooseFS as a direct fuse mount not exported via nfs.

This may indicate that this is not a problem for Gluster to solve. Evidence seems to indicate it to be a problem with how the knfs accesses fuse.

Comment 7 Josh Miller 2018-03-06 21:22:32 UTC
FWIW, I've also seen this with multiple versions of nfs-ganesha with both the XFS and GLUSTER FSAL's.

I used the following test:

echo "test" > file1; chmod 444 file1; cp file1 file2;
# that should work without error
# on error, we get: cp: closing `file2': Permission denied

Please note, this is not a timing issue with respect to when 'file1' was created. The copy will fail on any existing files that are mode 444 (or 440 or 400). I have not tried to avoid directory operations to allow directory caches to expire. That may be related, but I am not sure.

That fails on all of the following combos:

* nfs-ganesha 2.4.5 from CentOS Storage SIG (3.10), on CentOS 6, using GLUSTER FSAL (gluster 3.10.11) [see note below though]
* nfs-ganesha 2.5.2 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.3)
* nfs-ganesha 2.5.3 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.4)
* nfs-ganesha 2.5.3 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.5)
* nfs-ganesha 2.5.5 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.6)
* nfs-ganesha 2.5.5 from CentOS Storage SIG (3.13), on CentOS 6, using GLUSTER FSAL (gluster 3.13.2)
* nfs-ganesha 2.5.2 from CentOS Storage SIG (3.12), on CentOS 7, using XFS FSAL

Recently, this one DID work:
* nfs-ganesha 2.5.5 from CentOS Storage SIG (3.12), on CentOS 7, using XFS FSAL

Apparently, something was fixed in either the XFS FSAL or in ganesha itself. Seeing that work prompted me to re-test with GLUSTER FSAL, but it still doesn't work.


NOTE: Regarding ganesha 2.4.5, I'm fairly sure that did not work, but I didn't spend much time on that version. I ran into a different bug (presented when cvs prunes directories via "cvs up -P"), so I'm not 100% sure this bug was present in 2.4.5 or not.

Comment 8 Josh Miller 2018-03-06 23:53:53 UTC
(In reply to Josh Miller from comment #7)
> FWIW, I've also seen this with multiple versions of nfs-ganesha with both
> the XFS and GLUSTER FSAL's.

... nfs-ganesha-2.6.0 just got released to CentOS Storage SIG for gluster 3.13 on CentOS 6 today. No surprise, but it also exhibits the bug.

Comment 9 Rik Theys 2018-03-29 13:04:25 UTC
Hi,

I'm currently taking the RH236 course on gluster and was trying to reproduce this issue on the "Red Hat Gluster Storage" version of gluster.

I can not reproduce this issue with this version of nfs-ganesha.

The package versions of ganesha are:

nfs-ganesha-glsuter-2.2.0-12.el7rhgs.x86_64
glusterfs-ganesha-3.7.5-19.el7rhgs.x86_64
nfs-ganesha-2.2.0-12.el7rhgs.x86_64

So either this issue was introduced in a later version, or Red Hat has applied a patch to their nfs-ganesha version to fix this issue.

Regards,

Rik

Comment 10 Rik Theys 2018-03-29 13:05:46 UTC
(In reply to Rik Theys from comment #9)
> The package versions of ganesha are:
> 
> nfs-ganesha-glsuter-2.2.0-12.el7rhgs.x86_64
> glusterfs-ganesha-3.7.5-19.el7rhgs.x86_64
> nfs-ganesha-2.2.0-12.el7rhgs.x86_64
> 

The gluster package version is 3.7.5-19.el7rhgs

Comment 11 Raghavendra G 2018-05-25 05:49:00 UTC
RCA from Jiffin:
================
RCA:
The issue caused because open behind is trying to reopen the fd for write operation in ob_writev(). That open will fail because the file has permissions 444(readonly) which results the write failure. If we disable the open-behind then it will work as expected

Following is happening wrt to ganesha for cp file1 file2
client sends open adn read call on file1
client sends open call on file2
Ganesha first creates the file using glfs_h_creat, then after it get succeed, then ganesha opens global fd with glfs_h_open which sets open_behind_fd in ob_open call.

client sends setattr call to set permissions to 444 -- succeed

next clients sends write request,
if ganesha uses the fd opened during then it will be succeed. But in ob_writev it find there is an open_beind_fd, then it tries to wake the fd which internally sends open call, it will be failed because file has only read only permissions and hence write failed.

Don't much  know about internals of open behind. Requesting Du / Mlind to have look and provide their input for the bug.

Comment 12 Raghavendra G 2018-05-25 05:49:25 UTC
Its a bug in open-behind. NFS Ganesha is issuing a setattr call. Open-behind implements fsetattr, but not setattr. So, pending opens are not done before setattr resulting in open failures due to change in permissions. Component can be changed to Open-behind

Comment 13 Worker Ant 2018-05-25 07:02:37 UTC
REVIEW: https://review.gluster.org/20084 (performance/open-behind: open pending fds before permission change) posted (#1) for review on master by Raghavendra G

Comment 14 Worker Ant 2018-05-29 08:44:46 UTC
COMMIT: https://review.gluster.org/20084 committed in master by "Raghavendra G" <rgowdapp@redhat.com> with a commit message- performance/open-behind: open pending fds before permission change

setattr, posix-acl and selinux changes on a file can revoke permission
to open the file after permission changes. To prevent that, make sure
the pending fd is opened before winding down setattr or setxattr (for
posix-acl and selinux) calls.

Change-Id: Ib0b91795d286072e445190f9a1b3b1e9cd363282
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
fixes: bz#1405147

Comment 15 Shyamsundar 2018-10-23 15:06:00 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report.

glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.