If the zOS server can not handle this particular situation, how does it
handle unlinked opened files or open files which are the targets of
rename operations?

------- Comment From cvogan.com 2010-02-18 21:47 EDT-------
Hello Peter,  This is a tricky area of the z/OS NFS server legacy file systems (VSAM, PDS, PDSe and PS type datasets). The dataset format is ABCDEFGH.12345678.abcdefgh.12345678.   so you have multiple 8 char or less qualifiers with a max of 48 chars including the periods.
So its a known restriction when working with these datasets that you must close files before they are removed or risk an error during remove.

It seems to me that we can fix the sequencing here, but isn't there
going to be a general problem with the unlinked open files and
open files which are the targets of rename?

What if we changed the name of the silly renamed files to .nfsXXXXX?
Would that work?  It doesn't exactly match the filename pattern listed
in Comment #2...

------- Comment From cvogan.com 2010-02-19 11:06 EDT-------
I saw your comment about rename in your first post.  I do not fully understand how rename or move would be subject to the hidden .nfs file issue.  I ran tests with rename and move which both completed successfully. Granted the files were closed. As I said  The application must be designed to close files before removing them when working with MVS datasets.
As for the change you proposed.  It will not work. The reason is the name is still to long. I counted 9 characters including the period.  I am not looking for linux to resolve the length of the file used for a rename. But if we could prevent the client from getting to far ahead of its self and sending ops before getting replys, that  would help prevent this issue.   This rename issue of open files is not just limited to Linux.

The rename system call will remove the target filename if it exists.
If that file happened to be open already, then removing it would cause
it to be unaccessible.  Therefore, we use the sillyrename support to
keep the target file existing until it is no longer accessible via
the an open file.

Sorry, perhaps I wasn't be high level enough.  Is it possible to
construct a filename, beginning with .nfs, which would work for the
zOS server?

I believe that focusing on fixing this issue in the NFSv4 code is
not good.  There are two problems that I see.  One is that there are
specific tests in the testsuite which explicitly cause the sillyrename
code to be invoked.  I suppose that these test failures could be
ignored, and are probably already being ignored, but that does not
mean that the lack of correct support for sillyrename is a good thing.
Second, NFS clients, as a matter of normal operations, will occasionally
generate these sillyrename files.  If the server can't handle them,
then that server will not be generally useful as an NFS server.

------- Comment From cvogan.com 2010-02-19 12:13 EDT-------
It is possible to construct a name that would be compatible.
.nfs1234 would be 1 solution or we can follow the z/OS Naming convention.
.nfs1234.A1234567.B1234567

Note, The NFS server takes the leading period and maps it to an @ so it may store hidden files.

Each qualifier is a 1- to 8-character name. These characters may be alphanumeric, national ($, #, @), or the hyphen. The first character should be either alphabetic or national.

You can join qualifiers with periods. The maximum length of a data set name is as follows:
* Generally, 44 characters, including periods.

If you recall., The z/OS NFS server has 2 sides.  It has a Traditional MVS dataset side and a Unix Hierarchical file system Side. Only the Traditional MVS dataset side is affected.

------- Comment From cvogan.com 2010-03-02 11:13 EDT-------
I am going to try and recreate with RHEL6.0 Alpha3.

I have another question on the silly rename of a file.  What is the Minimum and Maximum length for a file that is renamed to .nfsxxxxxxxxxxxxxxxxxxxx ?
We are looking into ways to accommodate these hidden file renames.
The maximum rename file size is 36 (All chars/numbers including the leading period). Since it would be masked:
so .nfsxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(36) would be mapped to
@nfsxxxx.axxxxxxx.axxxxxxx.axxxxxxx.axxxxxxx(44)

The limits are as identified.  There are three parts, ".nfs", the
NFS_FILEID converted to hex, and then a static counter converted
to hex as well.  The fileid is used to reduce collisions when
multiple clients are creating sillyrename files in the same
directory.

Sorry, I only count 28 bytes, 4 + 16 + 8 in the current version
of the sillyrename code.  Where did the 36 come from?

A better solution might be use open(O_EXCL) to create the unique
name and then use rename.  By doing this, the NFS_FILEID portion
of the filename could be eliminated.  This would shrink the length
of the name down to 12, 4 + 8.

I suspect that this last solution may have a decent chance of
being accepted upstream.

------- Comment From cvogan.com 2010-03-02 12:15 EDT-------
The 36 came from the maximum length z/OS can accept  fit were to mask the silly rename object.  So if you you are always going to be close to 28 chars then z/OS may be able to make a change to handle these renames.  Other NFS client start at length 8 and then increment.
Unfortunately if NFS changed to length 12 (4+8) z/OS will still have to implement some sort of change to accommodate.  I will talk with development and see how likely it would be to mask these renamed files to something we can handle.

What if the fileID was length 64? Would that increase the size?

The size is _always_ 28.

The size of the fileid field is 64 bit.  The code to sprintf the
fileid specifies the minimum and maximum size to be printed and
those are 16 and 16, respectively.

The size of the counter is 32 bits.  It is handled exactly the
same way as the fileid.  It always prints 8 hex digits.

I don't see an easy change that would not require the zOS server
to be changed.  I also don't know of any NFS clients which would
not encounter a problem with the current zOS server.  Even Solaris
would eventually encounter a problem if enough open files were
either removed to renamed to.  It uses ".nfs" and then the counter,
although the counter is probably converted to decimal instead of
hex.

------- Comment From cvogan.com 2010-03-02 14:17 EDT-------
U are correct, We have have seen both AIX and Sun increment beyond 8 chars.
Since the confidence is high that Linux will not produce a silly rename >28 I will work on z/OS and see if we can get a change in place.

------- Comment From cvogan.com 2010-03-25 13:50 EDT-------
After some discussions with z/OS NFS team, It does not look like Redhat can change the silly rename behavior to address all instances on the z/OS NFS server.  But this is a known issue. One way to prevent silly renames from Linux is to have proper close/remove semantics which was addressed up stream a few years ago.

------- Comment From ffilz.com 2010-03-25 14:02 EDT-------
Ok, so it sounds like the way we want to proceed here is to have this patch included in RHEL 5.6:

http://git.linux-nfs.org/?p=trondmy/nfs-2.6.git&a=commitdiff&h=a49c3c7736a2e77931dabc5bc4a83fb4b2da013e

We understand silly renames will still occur in other situations, and that zOS will not be able to handle all silly rename variations.

------- Comment From ffilz.com 2010-04-06 16:00 EDT-------
I have backported the following patch:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a49c3c7736a2e77931dabc5bc4a83fb4b2da013e

In the process, to make the backport sane, I also backported this patch
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b39e625b6e75aa70e26c13f9378756bb5f2af032

Created attachment 404779 [details]
Backported patch to resolve issue


------- Comment (attachment only) From ffilz.com 2010-04-06 15:52 EDT-------

Thanks Frank, but the patch doesn't seem to apply cleanly to my RHEL5 tree. On what RHEL version did you base this?

------- Comment From ffilz.com 2010-04-13 16:57 EDT-------
I had done the patch against RHEL 5.4, we have updated to RHEL 5.5 and I redid the patch. We will test it later today or tomorow.

It looks like hunk1 for nfs4proc.c didn't apply because nfs4_call_async was changed to call rpc_new_task_wq instead of rpc_new_task.

Created attachment 406346 [details]
Patch backported for RHEL 5.5


------- Comment (attachment only) From ffilz.com 2010-04-13 16:52 EDT-------

Oops, probably should have mentioned that I backported some of the same patches for 2.6.18 and put them into the test kernels on my people page:

    http://people.redhat.com/jlayton/

...unless you see anything wrong with them, I'll probably go with those as they parallel the upstream fixes more closely. It looks like this also fixes an unrelated problem where the open_files list was being modified without holding the inode lock.

------- Comment From ffilz.com 2010-04-14 10:17 EDT-------
Oh, thanks Jeff.

Chris, could you verify Jeff's test kernel fixes your problem.

------- Comment From cvogan.com 2010-04-14 11:18 EDT-------
The latest patched kernel (2.6.18-194.el560829) on a RH EL 5.5 system was successful.

------- Comment From ffilz.com 2010-04-14 11:34 EDT-------
Great!

Jeff, please let us know when your patch set is accepted into the stream for RHEL 5.6.

Thanks everyone

I've got a newer version of this patch in my test kernels that fixes a possible regression in the old patch (the close would have been queued to rpciod rather than nfsiod).

It would be nice if IBM could test that patch too. The differences are minor, but I'd like to confirm that you don't see any problems from it.

------- Comment From ffilz.com 2010-05-03 14:00 EDT-------
Jeff, your kernels here:


have this update correct?

That's correct.

------- Comment From cvogan.com 2010-05-04 09:49 EDT-------
I applied kernel  2.6.18-197.el5.jtltest.101.
I was unable to re-recreate the silly rename issue with a z/OS NFS server that was under heavy load. It looks like the test kernel from Jeff also resolves the issue.

in kernel-2.6.18-207.el5
You can download this test kernel from http://people.redhat.com/jwilson/el5

Detailed testing feedback is always welcomed.

------- Comment From cvogan.com 2010-07-19 23:38 EDT-------
Hello I just tried to recreate the failing case with kernel:
Linux reddy1 2.6.18-207.el5 #1 SMP Thu Jul 15 18:22:40 EDT 2010 i686 i686 i386 GNU/Linux

My test completed successfully.

I wrote a simple test case to test the failing condition to kernel:
2.6.18-194.el5PAE

openclose.c
#include<stdio.h>
#include<stdlib.h>
#include<fcntl.h>
#include<string.h>
#include<ctype.h>
#include<time.h>
#include<errno.h>

int main(int argc, char *argv[])
{
int fd=0;                        /*file descriptor*/
char startDir[256];
char mntpt[256];
char inputfile[256];

if (argc != 3) {
printf("Usage: %s <mount point> <data file>\n", argv[0]);
exit(1);
}
strcpy(mntpt, argv[1]);
strcpy(inputfile, argv[2]);
if (chdir(mntpt) < 0) {
printf("Could not Change dir to mount point\n");
exit(1);
}
if ((fd=fopen( inputfile, "w+")) == 0) {
printf("ERROR NO. %d\n",fd);
printf("FAILED on open file %s\n",inputfile);
exit(1);
}
if ((fclose(fd)) < 0 ) {
printf("Could Not Close file %s\n",inputfile);
exit(1);
}
printf("I just tried to remove file: %s\n",inputfile);
if ((remove(inputfile)) < 0 ) {
printf("Failed to remove file %s\n",inputfile);
exit(1);
}
printf("I just tried to close file: %s\n",inputfile);
}

I just wrapped this C program around a looping shell script to run it many times in quick succession.

*** Bug 439093 has been marked as a duplicate of this bug. ***

test against comment #33 on RHEL5.6-Server-20101014.0 on i386 and x86_64, run it in loop and can completed successfully.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, Connectathon test cases performed on a z/OS NFSv4 server were regularly failing. While the file was being closed prior to the unlink call, the client did not wait for the close to complete before proceeding. This caused it to perform an inappropriate rename instead of unlinking the file, even though it was not required. With this update, removals on NFSv4 mounts should wait for outstanding close calls to complete before proceeding.

This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux minor release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux Update minor release.   
While it is a goal to include this enhancement in the next minor release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next minor release pending the next phase of actual 
code integration and successful Red Hat and partner testing.

------- Comment From sglass.com 2010-11-30 15:40 EDT-------
This feature has been verified.  Thanks

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html