Created attachment 483140 [details] Script to append data to the file Description of problem: We have a ksh script that appends data to a world-writable file, which is stored on an NFSv4 mount. When we run the script, we get an error stating the file does not exist and that it can not be created (which is true as there are no write permissions on the directory). But the file DOES EXIST. If we stat the file in question, the information shown is correct. Running the command again after the stat no longer returns an error and the line is appended. The server is a RHEL6 server running 2.6.32-71.18.1.el6.x86_64. The client is running the RHEL 5.6 kernel, but we also have this problem with 5.5 clients. Same problem on i386 and x86_64. The directory in which the world-writable file is stored is automounted. Version-Release number of selected component (if applicable): All RHEL 5.5 and 5.6 kernels. Haven't tested CentOS 4 kernels or pre-5.5 kernels. How reproducible: Almost always in our environment. Steps to Reproduce: 1. set-up a world-writable file on an NFSv4 share and make sure the directory in which this file is stored is automounted on the client 2. run the script in attach 3. Actual results: Script shows error message: /freeware/gcc/linux-x86_64/bin/accountlog: cannot create [Permission denied] This erorr is reproducable until a stat of the file is done. Once a 'stat /freeware/gcc/linux-x86_64/bin/accountlog' shows the correct info, the script can be run without error. Expected results: No error and the line appended to the world-writable file. Additional info: We also see strange errors like this with other (not-ksh) scripts, so it's probably not ksh related.
Created attachment 483141 [details] strace of the command when the error is shown
Created attachment 483142 [details] strace output after running the stat command
I tested writing to an NFSv4 mount which is manually mounted (not autofs mounted) and the problem is the same.
I tried this with a Debian squeeze 2.6.32 version 2.6.32-30 as the server and it also has this problem. Also tried with the 2.6.38-rc6-amd64 2.6.38~rc6-1~experimental.1 kernel and also have this bug with that kernel.
I tried it with an ext3 file system on the server, but this has the same behaviour.
I filed a bug in the kernel bugzilla as this seems to be an upstream bug. https://bugzilla.kernel.org/show_bug.cgi?id=30862
I could easily reproduce this issue. The problem is caused by the O_CREAT flag which results in an OPEN call being made to the NFS server with Open type set to OPEN4_CREATE and Open mode set to UNCHECKED. Since the user opening this file cannot actually create a file on the server, this call fails with a EACCES. This is a NFS server bug. From RFC 3530[page 172], for Open Type OPEN4_CREATE and open mode UNCHECKED. UNCHECKED means that the file should be created if a file of that name does not exist and encountering an existing regular file of that name is not an error. For this type of create, createattrs specifies the initial set of attributes for the file. The set of attributes may include any writable attribute valid for regular files. When an UNCHECKED create encounters an existing file, the attributes specified by createattrs are not used, except that when an size of zero is specified, the existing file is truncated. If GUARDED is specified, the server checks for the presence of a duplicate object by name before performing the create. If a duplicate exists, an error of NFS4ERR_EXIST is returned as the status. If the object does not exist, the request is performed as described for UNCHECKED. For each of these cases (UNCHECKED and GUARDED) where the operation is successful, the server will return to the client an attribute mask signifying which attributes were successfully set for the object. Since open mode is set to UNCHECKED, the NFS server when encountering a regular file, it should open the existing file and not treat the CREATE call as an error.
Created attachment 491465 [details] Simple program to demonstrate the problem. On the NFS server: # cat /etc/exports /exports *(rw,no_root_squash) NFS exports contains a directory with mode set to 555 to prevent non priviledged users from writing to it. # ls -ld /exports/test dr-xr-xr-x. 2 root root 4096 Apr 11 13:48 /exports/test # ls -l /exports/test/a -rwxrwxrwx. 1 root root 29 Apr 11 13:49 /exports/test/a Mount this share on a NFS client and run the command simple_open to open the file /exports/test/a on the NFS server [test1@vm130-31 ~]$ ./simple_open /mnt/test/a open: Permission denied
Created attachment 491466 [details] tcpdump from the above attached reproducer simple_open.
The following call successful recreated the problem. open( argv[1], O_CREAT | O_RDWR ) and open failed with EACCES while this was modified to the following, it opened without any problems open( argv[1], O_RDWR ) Once the file had been successfully opened, we no longer can reproduce this issue.
The problem is seen because of the fh_verify call to check for create permissions in nfsd_create_v3(). A simple patch to check for permissions only if the file doesn't exist fixes the problem. diff -up nfsd/vfs.c.orig nfsd/vfs.c --- nfsd/vfs.c.orig 2011-04-12 15:07:13.450652348 +0100 +++ nfsd/vfs.c 2011-04-12 15:42:00.836567891 +0100 @@ -1394,7 +1394,7 @@ nfsd_create_v3(struct svc_rqst *rqstp, s goto out; if (!(iap->ia_valid & ATTR_MODE)) iap->ia_mode = 0; - err = fh_verify(rqstp, fhp, S_IFDIR, NFSD_MAY_CREATE); + err = fh_verify(rqstp, fhp, S_IFDIR, NFSD_MAY_EXEC); if (err) goto out; @@ -1415,6 +1415,13 @@ nfsd_create_v3(struct svc_rqst *rqstp, s host_err = PTR_ERR(dchild); if (IS_ERR(dchild)) goto out_nfserr; + + /* If file doesn't exist, check for permissions to create one */ + if(!dchild->d_inode){ + err = fh_verify(rqstp, fhp, S_IFDIR, NFSD_MAY_CREATE); + if (err) + goto out; + } err = fh_compose(resfhp, fhp->fh_export, dchild, fhp); if (err)
I recompiled the RHEL6 kernel with this patch applied and it seems to fix the problem. The kernel I recompiled is 2.6.32-71.24.1.el6. Will this patch make it upstream, the upstream stable releases and in a RHEL4/5/6 update?
Rik, We aim to have all out patches first accepted upstream before we backport it to RHEL kernels. I'll post the patch upstream and paste the url here. Sachin Prabhu
Sachin, is there some reason the patch hasn't shown up on linux-nfs@vger?
Bruce, I just wanted to run it bye you before sending it up. I'll post it shortly. Sachin Prabhu
http://thread.gmane.org/gmane.linux.nfs/40010
The fix is now in Linus' tree. Can the fix be backported to the RHEL kernels?
Yep, upstream as 1574dff8996ab1ed92c09012f8038b5566fce313, thanks Sachin, and applies with minor fixups to rhel5 and 6.
http://post-office.corp.redhat.com/archives/rhkernel-list/2011-May/msg00119.html
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Patch(es) available in kernel-2.6.18-282.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed.
Test result: reproduce ====== [test@hp-p6100z-01 ~]$ ./bz683372 /mnt/test/test/test open: Permission denied [test@hp-p6100z-01 ~]$ uname -a Linux hp-p6100z-01.lab.bos.redhat.com 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [test@hp-p6100z-01 ~]$ exit exit [root@hp-p6100z-01 ~]# ll /mnt/test/test -da dr-xr-xr-x 2 root root 4096 Aug 24 01:10 /mnt/test/test [root@hp-p6100z-01 ~]# ll /mnt/test/test -a total 20 dr-xr-xr-x 2 root root 4096 Aug 24 01:10 . drwxrwxrwt 5 root root 4096 Aug 24 01:23 .. -rwxrwxrwx 1 root root 0 Aug 24 01:10 test [root@hp-p6100z-01 ~]# mount | grep /mnt/test localhost:/ on /mnt/test type nfs4 (rw,addr=127.0.0.1) verify ====== [root@hp-p6100z-01 ~]# mount | grep /mnt/test localhost:/ on /mnt/test type nfs4 (rw,addr=127.0.0.1) [root@hp-p6100z-01 ~]# uname -a Linux hp-p6100z-01.lab.bos.redhat.com 2.6.18-282.el5 #1 SMP Mon Aug 22 15:14:01 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@hp-p6100z-01 ~]# ll /mnt/test/test/test -rwxrwxrwx 1 root root 0 Aug 24 01:10 /mnt/test/test/test [root@hp-p6100z-01 ~]# ll /mnt/test/test -d dr-xr-xr-x 2 root root 4096 Aug 24 01:10 /mnt/test/test [root@hp-p6100z-01 ~]# su test [test@hp-p6100z-01 root]$ cd [test@hp-p6100z-01 ~]$ ./bz683372 /mnt/test/test/test Success
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0150.html