Description of problem: When using a fuse-mounted glusterfs volume, an error is found if the owner of a file (that is not root) tries to write in its own file when it has read-only permissions. The problem does not exist if the user is root. Version-Release number of selected component (if applicable): All versions tested (glusterfs-3.7.{4-16}, glusterfs-3.8.{4-5}) How reproducible: Always Steps to Reproduce: 1. Fuse mount a glusterfs volume to be the home of the user, cd to home dir and check that the file we will use for testing does not exist: user@computer:~$ ls -l 444.txt ls: cannot access 444.txt: No such file or directory user@computer:~$ 2. Execute command (error comes at echo, not before): user@computer:~$ rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo test > 444.txt -bash: 444.txt: Permission denied 3. Check result: File is empty. user@computer:~$ ls -l 444.txt -r--r--r-- 1 user user 0 dic 15 2016 444.txt user@computer:~$ rm -f 444.txt user@computer:~$ 4. Check that the problem dissapears with root privileges: user@computer:~$ sudo su root@computer:/home/user# rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo test > 444.txt root@computer:/home/user# ls -l 444.txt -r--r--r-- 1 root root 5 dic 15 2016 444.txt root@computer:/home/user# rm -f 444.txt root@computer:/home/user# Actual results: A user who owns a file that is read-only cannot write into it. Expected results: A user who owns a file that is read-only should be able to write into it even if it is read-only. Additional info: Special thanks to Bruce Fields and other components of nfs-kernel-server team because initially we thought it was their fault. The same to Soumya Koduri and Miklos Szeredi, that helped to confirm the problem. See full previous thread on nfs-kernel-server list with history about the issue: http://marc.info/?t=144801803400001 (for posterity's sake, permanent link: http://marc.info/?i=862ef8b0-61ee-8b7e-a81c-fcdc5a726e37@dit.upm.es).
Can u change the product into glusterfs and component into access-control. IMO it is accepted behavior. Posix compliant file system(xfs) behaves in a similar way. [jiffin@jiffin17 ~]$ touch file [jiffin@jiffin17 ~]$ chmod 444 file [jiffin@jiffin17 ~]$ echo read > file bash: file: Permission denied [root@jiffin17 ~]# touch file [root@jiffin17 ~]# chmod 444 file [root@jiffin17 ~]# echo read > file GlusterFS being posix compliant, it should behave like this
Jiffin, This is not for the WRITE after OPEN... but while creating a file, file_owner should get the write permissions. Try this particular case - #rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt It works fine on fuse mount but not when that fuse mount is exported via kernel-nfs and mounted using nfs client.
and agree that it should have been filed against upstream glusterfs component.
Sorry to all, I have to apologize because I tried to super-simplify the problem in the explanation without remembering properly previous tests made and possible issues discarded... The origin of this problem is about copying a read-only file, not about writing data on it. Tell me if you think maybe is better to file a new bug or to explain it again in this bug report. If the later, the explanation would be the following (based on http://marc.info/?l=linux-nfs&m=147932042025835): Description of problem: When you export a glusterfs fuse-mounted volume through a nfs-kernel-server, if the nfs-client mounting it tries to copy an owned read-only file -not being root user-, a permission denied error shows up at the client side and it is recorded at the glusterfs logs. If the user is root no problem exist. Version-Release number of selected component (if applicable): All versions tested (glusterfs-3.7.{4-16}, glusterfs-3.8.{4-5}) How reproducible: Always Steps to Reproduce: The setup is: nfs client---NFSv3--->knfsd---FUSE-->gluster The tests are: # At the knfsd server: root@cuentas03-lab:/etc# cat /proc/mounts [...] recipiente6hd:/home-lab-3.tcp /home-3-old fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0 root@cuentas03-lab:/etc# # At the nfs-client: cdc@l056:~$ cat /proc/mounts | grep "/home/cdc" cuentas03:/home-3-old/cdc /home/cdc nfs rw,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=138.4.30.18,mountvers=3,mountport=49531,mountproto=tcp,fsc,local_lock=all,addr=138.4.30.18 0 0 cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt cp: failed to close ‘kk.txt’: Permission denied -r--r--r-- 1 cdc admincdc 7 nov 16 2016 444.txt -r--r--r-- 1 cdc admincdc 0 nov 16 2016 kk.txt cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ cdc@l056:~$ sudo su root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt -r--r--r-- 1 root root 7 nov 16 2016 444.txt -r--r--r-- 1 root root 7 nov 16 2016 kk.txt root@l056:/home/cdc# exit cdc@l056:~$ Actual results: As a nfs-client of a fuse-mounted glusterfs volume, a user who owns a file that is read-only cannot copy it to a new file. Expected results: As a nfs-client of a fuse-mounted glusterfs volume, a user who owns a file that is read-only should be able to copy it even if it is read-only. Hope this helps to clarify. Sorry for the noise.
We are seeing the same issue but this time on MooseFS exported as nfs via kernel nfs. The same test works fine when accessing MooseFS as a direct fuse mount not exported via nfs. This may indicate that this is not a problem for Gluster to solve. Evidence seems to indicate it to be a problem with how the knfs accesses fuse.
FWIW, I've also seen this with multiple versions of nfs-ganesha with both the XFS and GLUSTER FSAL's. I used the following test: echo "test" > file1; chmod 444 file1; cp file1 file2; # that should work without error # on error, we get: cp: closing `file2': Permission denied Please note, this is not a timing issue with respect to when 'file1' was created. The copy will fail on any existing files that are mode 444 (or 440 or 400). I have not tried to avoid directory operations to allow directory caches to expire. That may be related, but I am not sure. That fails on all of the following combos: * nfs-ganesha 2.4.5 from CentOS Storage SIG (3.10), on CentOS 6, using GLUSTER FSAL (gluster 3.10.11) [see note below though] * nfs-ganesha 2.5.2 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.3) * nfs-ganesha 2.5.3 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.4) * nfs-ganesha 2.5.3 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.5) * nfs-ganesha 2.5.5 from CentOS Storage SIG (3.12), on CentOS 6, using GLUSTER FSAL (gluster 3.12.6) * nfs-ganesha 2.5.5 from CentOS Storage SIG (3.13), on CentOS 6, using GLUSTER FSAL (gluster 3.13.2) * nfs-ganesha 2.5.2 from CentOS Storage SIG (3.12), on CentOS 7, using XFS FSAL Recently, this one DID work: * nfs-ganesha 2.5.5 from CentOS Storage SIG (3.12), on CentOS 7, using XFS FSAL Apparently, something was fixed in either the XFS FSAL or in ganesha itself. Seeing that work prompted me to re-test with GLUSTER FSAL, but it still doesn't work. NOTE: Regarding ganesha 2.4.5, I'm fairly sure that did not work, but I didn't spend much time on that version. I ran into a different bug (presented when cvs prunes directories via "cvs up -P"), so I'm not 100% sure this bug was present in 2.4.5 or not.
(In reply to Josh Miller from comment #7) > FWIW, I've also seen this with multiple versions of nfs-ganesha with both > the XFS and GLUSTER FSAL's. ... nfs-ganesha-2.6.0 just got released to CentOS Storage SIG for gluster 3.13 on CentOS 6 today. No surprise, but it also exhibits the bug.
Hi, I'm currently taking the RH236 course on gluster and was trying to reproduce this issue on the "Red Hat Gluster Storage" version of gluster. I can not reproduce this issue with this version of nfs-ganesha. The package versions of ganesha are: nfs-ganesha-glsuter-2.2.0-12.el7rhgs.x86_64 glusterfs-ganesha-3.7.5-19.el7rhgs.x86_64 nfs-ganesha-2.2.0-12.el7rhgs.x86_64 So either this issue was introduced in a later version, or Red Hat has applied a patch to their nfs-ganesha version to fix this issue. Regards, Rik
(In reply to Rik Theys from comment #9) > The package versions of ganesha are: > > nfs-ganesha-glsuter-2.2.0-12.el7rhgs.x86_64 > glusterfs-ganesha-3.7.5-19.el7rhgs.x86_64 > nfs-ganesha-2.2.0-12.el7rhgs.x86_64 > The gluster package version is 3.7.5-19.el7rhgs
RCA from Jiffin: ================ RCA: The issue caused because open behind is trying to reopen the fd for write operation in ob_writev(). That open will fail because the file has permissions 444(readonly) which results the write failure. If we disable the open-behind then it will work as expected Following is happening wrt to ganesha for cp file1 file2 client sends open adn read call on file1 client sends open call on file2 Ganesha first creates the file using glfs_h_creat, then after it get succeed, then ganesha opens global fd with glfs_h_open which sets open_behind_fd in ob_open call. client sends setattr call to set permissions to 444 -- succeed next clients sends write request, if ganesha uses the fd opened during then it will be succeed. But in ob_writev it find there is an open_beind_fd, then it tries to wake the fd which internally sends open call, it will be failed because file has only read only permissions and hence write failed. Don't much know about internals of open behind. Requesting Du / Mlind to have look and provide their input for the bug.
Its a bug in open-behind. NFS Ganesha is issuing a setattr call. Open-behind implements fsetattr, but not setattr. So, pending opens are not done before setattr resulting in open failures due to change in permissions. Component can be changed to Open-behind
REVIEW: https://review.gluster.org/20084 (performance/open-behind: open pending fds before permission change) posted (#1) for review on master by Raghavendra G
COMMIT: https://review.gluster.org/20084 committed in master by "Raghavendra G" <rgowdapp> with a commit message- performance/open-behind: open pending fds before permission change setattr, posix-acl and selinux changes on a file can revoke permission to open the file after permission changes. To prevent that, make sure the pending fd is opened before winding down setattr or setxattr (for posix-acl and selinux) calls. Change-Id: Ib0b91795d286072e445190f9a1b3b1e9cd363282 Signed-off-by: Raghavendra G <rgowdapp> fixes: bz#1405147
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report. glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html [2] https://www.gluster.org/pipermail/gluster-users/