Bug 701825

+++ This bug was initially created as a clone of Bug #683372 +++

Created attachment 483140 [details]
Script to append data to the file

Description of problem:

We have a ksh script that appends data to a world-writable file, which is stored on an NFSv4 mount. When we run the script, we get an error stating the file does not exist and that it can not be created (which is true as there are no write permissions on the directory).

But the file DOES EXIST. If we stat the file in question, the information shown is correct. Running the command again after the stat no longer returns an error and the line is appended.

The server is a RHEL6 server running 2.6.32-71.18.1.el6.x86_64.

The client is running the RHEL 5.6 kernel, but we also have this problem with 5.5 clients. Same problem on i386 and x86_64.

The directory in which the world-writable file is stored is automounted.

Version-Release number of selected component (if applicable):

All RHEL 5.5 and 5.6 kernels. Haven't tested CentOS 4 kernels or pre-5.5 kernels.


How reproducible:

Almost always in our environment.

Steps to Reproduce:
1. set-up a world-writable file on an NFSv4 share and make sure the directory in which this file is stored is automounted on the client
2. run the script in attach
3. 
  
Actual results:

Script shows error message:

/freeware/gcc/linux-x86_64/bin/accountlog: cannot create [Permission denied]

This erorr is reproducable until a stat of the file is done. Once a 'stat /freeware/gcc/linux-x86_64/bin/accountlog' shows the correct info, the script can be run without error.


Expected results:

No error and the line appended to the world-writable file.

Additional info:

We also see strange errors like this with other (not-ksh) scripts, so it's probably not ksh related.

--- Additional comment from rik.theys.ac.be on 2011-03-09 04:55:21 EST ---

Created attachment 483141 [details]
strace of the command when the error is shown

--- Additional comment from rik.theys.ac.be on 2011-03-09 04:55:47 EST ---

Created attachment 483142 [details]
strace output after running the stat command

--- Additional comment from rik.theys.ac.be on 2011-03-09 05:09:54 EST ---

I tested writing to an NFSv4 mount which is manually mounted (not autofs mounted) and the problem is the same.

--- Additional comment from rik.theys.ac.be on 2011-03-09 08:34:06 EST ---

I tried this with a Debian squeeze 2.6.32 version 2.6.32-30 as the server and it also has this problem. Also tried with the 2.6.38-rc6-amd64 2.6.38~rc6-1~experimental.1 kernel and also have this bug with that kernel.

--- Additional comment from rik.theys.ac.be on 2011-03-09 09:33:34 EST ---

I tried it with an ext3 file system on the server, but this has the same behaviour.

--- Additional comment from rik.theys.ac.be on 2011-03-10 07:29:40 EST ---

I filed a bug in the kernel bugzilla as this seems to be an upstream bug.

https://bugzilla.kernel.org/show_bug.cgi?id=30862

--- Additional comment from sprabhu on 2011-04-12 07:57:56 EDT ---

I could easily reproduce this issue.

The problem is caused by the O_CREAT flag which results in an OPEN call being made to the NFS server with Open type set to OPEN4_CREATE and Open mode set to UNCHECKED. Since the user opening this file cannot actually create a file on the server, this call fails with a EACCES. 

This is a NFS server bug. From RFC 3530[page 172], for Open Type OPEN4_CREATE and open mode UNCHECKED.

   UNCHECKED means that the file should be created if a file of that
   name does not exist and encountering an existing regular file of that
   name is not an error.  For this type of create, createattrs specifies
   the initial set of attributes for the file.  The set of attributes
   may include any writable attribute valid for regular files.  When an
   UNCHECKED create encounters an existing file, the attributes
   specified by createattrs are not used, except that when an size of
   zero is specified, the existing file is truncated.  If GUARDED is
   specified, the server checks for the presence of a duplicate object
   by name before performing the create.  If a duplicate exists, an
   error of NFS4ERR_EXIST is returned as the status.  If the object does
   not exist, the request is performed as described for UNCHECKED.  For
   each of these cases (UNCHECKED and GUARDED) where the operation is
   successful, the server will return to the client an attribute mask
   signifying which attributes were successfully set for the object.

Since open mode is set to UNCHECKED, the NFS server when encountering a regular file, it should open the existing file and not treat the CREATE call as an error.

--- Additional comment from sprabhu on 2011-04-12 08:04:20 EDT ---

Created attachment 491465 [details]
Simple program to demonstrate the problem.

On the NFS server:

# cat /etc/exports 
/exports *(rw,no_root_squash)

NFS exports contains a directory with mode set to 555 to prevent non priviledged users from writing to it.

# ls -ld /exports/test
dr-xr-xr-x. 2 root root 4096 Apr 11 13:48 /exports/test
# ls -l /exports/test/a
-rwxrwxrwx. 1 root root 29 Apr 11 13:49 /exports/test/a


Mount this share on a NFS client and run the command simple_open to open the file /exports/test/a on the NFS server

[test1@vm130-31 ~]$ ./simple_open /mnt/test/a
open: Permission denied

--- Additional comment from sprabhu on 2011-04-12 08:04:58 EDT ---

Created attachment 491466 [details]
tcpdump from the above attached reproducer simple_open.

--- Additional comment from sprabhu on 2011-04-12 08:05:40 EDT ---

The following call successful recreated the problem.
open( argv[1], O_CREAT | O_RDWR  ) 
and open failed with EACCES
while this was modified to the following, it opened without any problems
open( argv[1], O_RDWR  )

Once the file had been successfully opened, we no longer can reproduce this issue.

--- Additional comment from sprabhu on 2011-04-12 11:06:33 EDT ---

The problem is seen because of the fh_verify call to check for create permissions in nfsd_create_v3(). A simple patch to check for permissions only if the file doesn't exist fixes the problem.


diff -up nfsd/vfs.c.orig nfsd/vfs.c
--- nfsd/vfs.c.orig	2011-04-12 15:07:13.450652348 +0100
+++ nfsd/vfs.c	2011-04-12 15:42:00.836567891 +0100
@@ -1394,7 +1394,7 @@ nfsd_create_v3(struct svc_rqst *rqstp, s
 		goto out;
 	if (!(iap->ia_valid & ATTR_MODE))
 		iap->ia_mode = 0;
-	err = fh_verify(rqstp, fhp, S_IFDIR, NFSD_MAY_CREATE);
+	err = fh_verify(rqstp, fhp, S_IFDIR, NFSD_MAY_EXEC);
 	if (err)
 		goto out;
 
@@ -1415,6 +1415,13 @@ nfsd_create_v3(struct svc_rqst *rqstp, s
 	host_err = PTR_ERR(dchild);
 	if (IS_ERR(dchild))
 		goto out_nfserr;
+	
+	/* If file doesn't exist, check for permissions to create one */
+	if(!dchild->d_inode){
+		err = fh_verify(rqstp, fhp, S_IFDIR, NFSD_MAY_CREATE);
+		if (err)
+			goto out;
+	}
 
 	err = fh_compose(resfhp, fhp->fh_export, dchild, fhp);
 	if (err)

--- Additional comment from rik.theys.ac.be on 2011-04-14 07:55:54 EDT ---

I recompiled the RHEL6 kernel with this patch applied and it seems to fix the problem. The kernel I recompiled is 2.6.32-71.24.1.el6.

Will this patch make it upstream, the upstream stable releases and in a RHEL4/5/6 update?

--- Additional comment from sprabhu on 2011-04-14 08:16:24 EDT ---

Rik,

We aim to have all out patches first accepted upstream before we backport it to RHEL kernels. I'll post the patch upstream and paste the url here.

Sachin Prabhu

--- Additional comment from bfields on 2011-04-19 16:45:32 EDT ---

Sachin, is there some reason the patch hasn't shown up on linux-nfs@vger?

--- Additional comment from sprabhu on 2011-04-20 05:33:56 EDT ---

Bruce,

I just wanted to run it bye you before sending it up. I'll post it shortly.

Sachin Prabhu

--- Additional comment from sprabhu on 2011-04-20 08:10:48 EDT ---

http://thread.gmane.org/gmane.linux.nfs/40010

--- Additional comment from rik.theys.ac.be on 2011-04-22 08:27:25 EDT ---

The fix is now in Linus' tree. Can the fix be backported to the RHEL kernels?

--- Additional comment from bfields on 2011-05-03 20:02:23 EDT ---

Yep, upstream as 1574dff8996ab1ed92c09012f8038b5566fce313, thanks Sachin, and applies with minor fixups to rhel5 and 6.

--- Additional comment from bfields on 2011-05-03 20:56:20 EDT ---

http://post-office.corp.redhat.com/archives/rhkernel-list/2011-May/msg00119.html