Bug 565974
Summary: | [5.6 FEAT] NFSv4 remove does not wait for close. Silly rename | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 5 | Reporter: | IBM Bug Proxy <bugproxy> | ||||||
Component: | kernel | Assignee: | Jeff Layton <jlayton> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Red Hat Kernel QE team <kernel-qe> | ||||||
Severity: | medium | Docs Contact: | |||||||
Priority: | medium | ||||||||
Version: | 5.6 | CC: | chrisvogan, dhoward, jburke, jjarvis, jlayton, jpirko, nobody+PNT0273897, rwheeler, sbest, steved, yanwang | ||||||
Target Milestone: | rc | Keywords: | FutureFeature, ZStream | ||||||
Target Release: | 5.6 | ||||||||
Hardware: | All | ||||||||
OS: | All | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Enhancement | |||||||
Doc Text: |
Previously, Connectathon test cases performed on a z/OS NFSv4 server were regularly failing. While the file was being closed prior to the unlink call, the client did not wait for the close to complete before proceeding. This caused it to perform an inappropriate rename instead of unlinking the file, even though it was not required. With this update, removals on NFSv4 mounts should wait for outstanding close calls to complete before proceeding.
|
Story Points: | --- | ||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2011-01-13 21:06:21 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 531800, 557597, 642628, 702355 | ||||||||
Attachments: |
|
Description
IBM Bug Proxy
2010-02-16 20:31:31 UTC
If the zOS server can not handle this particular situation, how does it handle unlinked opened files or open files which are the targets of rename operations? ------- Comment From cvogan.com 2010-02-18 21:47 EDT------- Hello Peter, This is a tricky area of the z/OS NFS server legacy file systems (VSAM, PDS, PDSe and PS type datasets). The dataset format is ABCDEFGH.12345678.abcdefgh.12345678. so you have multiple 8 char or less qualifiers with a max of 48 chars including the periods. So its a known restriction when working with these datasets that you must close files before they are removed or risk an error during remove. It seems to me that we can fix the sequencing here, but isn't there going to be a general problem with the unlinked open files and open files which are the targets of rename? What if we changed the name of the silly renamed files to .nfsXXXXX? Would that work? It doesn't exactly match the filename pattern listed in Comment #2... ------- Comment From cvogan.com 2010-02-19 11:06 EDT------- I saw your comment about rename in your first post. I do not fully understand how rename or move would be subject to the hidden .nfs file issue. I ran tests with rename and move which both completed successfully. Granted the files were closed. As I said The application must be designed to close files before removing them when working with MVS datasets. As for the change you proposed. It will not work. The reason is the name is still to long. I counted 9 characters including the period. I am not looking for linux to resolve the length of the file used for a rename. But if we could prevent the client from getting to far ahead of its self and sending ops before getting replys, that would help prevent this issue. This rename issue of open files is not just limited to Linux. The rename system call will remove the target filename if it exists. If that file happened to be open already, then removing it would cause it to be unaccessible. Therefore, we use the sillyrename support to keep the target file existing until it is no longer accessible via the an open file. Sorry, perhaps I wasn't be high level enough. Is it possible to construct a filename, beginning with .nfs, which would work for the zOS server? I believe that focusing on fixing this issue in the NFSv4 code is not good. There are two problems that I see. One is that there are specific tests in the testsuite which explicitly cause the sillyrename code to be invoked. I suppose that these test failures could be ignored, and are probably already being ignored, but that does not mean that the lack of correct support for sillyrename is a good thing. Second, NFS clients, as a matter of normal operations, will occasionally generate these sillyrename files. If the server can't handle them, then that server will not be generally useful as an NFS server. ------- Comment From cvogan.com 2010-02-19 12:13 EDT------- It is possible to construct a name that would be compatible. .nfs1234 would be 1 solution or we can follow the z/OS Naming convention. .nfs1234.A1234567.B1234567 Note, The NFS server takes the leading period and maps it to an @ so it may store hidden files. Each qualifier is a 1- to 8-character name. These characters may be alphanumeric, national ($, #, @), or the hyphen. The first character should be either alphabetic or national. You can join qualifiers with periods. The maximum length of a data set name is as follows: * Generally, 44 characters, including periods. If you recall., The z/OS NFS server has 2 sides. It has a Traditional MVS dataset side and a Unix Hierarchical file system Side. Only the Traditional MVS dataset side is affected. ------- Comment From cvogan.com 2010-03-02 11:13 EDT------- I am going to try and recreate with RHEL6.0 Alpha3. I have another question on the silly rename of a file. What is the Minimum and Maximum length for a file that is renamed to .nfsxxxxxxxxxxxxxxxxxxxx ? We are looking into ways to accommodate these hidden file renames. The maximum rename file size is 36 (All chars/numbers including the leading period). Since it would be masked: so .nfsxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(36) would be mapped to @nfsxxxx.axxxxxxx.axxxxxxx.axxxxxxx.axxxxxxx(44) The limits are as identified. There are three parts, ".nfs", the NFS_FILEID converted to hex, and then a static counter converted to hex as well. The fileid is used to reduce collisions when multiple clients are creating sillyrename files in the same directory. Sorry, I only count 28 bytes, 4 + 16 + 8 in the current version of the sillyrename code. Where did the 36 come from? A better solution might be use open(O_EXCL) to create the unique name and then use rename. By doing this, the NFS_FILEID portion of the filename could be eliminated. This would shrink the length of the name down to 12, 4 + 8. I suspect that this last solution may have a decent chance of being accepted upstream. ------- Comment From cvogan.com 2010-03-02 12:15 EDT------- The 36 came from the maximum length z/OS can accept fit were to mask the silly rename object. So if you you are always going to be close to 28 chars then z/OS may be able to make a change to handle these renames. Other NFS client start at length 8 and then increment. Unfortunately if NFS changed to length 12 (4+8) z/OS will still have to implement some sort of change to accommodate. I will talk with development and see how likely it would be to mask these renamed files to something we can handle. What if the fileID was length 64? Would that increase the size? The size is _always_ 28. The size of the fileid field is 64 bit. The code to sprintf the fileid specifies the minimum and maximum size to be printed and those are 16 and 16, respectively. The size of the counter is 32 bits. It is handled exactly the same way as the fileid. It always prints 8 hex digits. I don't see an easy change that would not require the zOS server to be changed. I also don't know of any NFS clients which would not encounter a problem with the current zOS server. Even Solaris would eventually encounter a problem if enough open files were either removed to renamed to. It uses ".nfs" and then the counter, although the counter is probably converted to decimal instead of hex. ------- Comment From cvogan.com 2010-03-02 14:17 EDT------- U are correct, We have have seen both AIX and Sun increment beyond 8 chars. Since the confidence is high that Linux will not produce a silly rename >28 I will work on z/OS and see if we can get a change in place. ------- Comment From cvogan.com 2010-03-25 13:50 EDT------- After some discussions with z/OS NFS team, It does not look like Redhat can change the silly rename behavior to address all instances on the z/OS NFS server. But this is a known issue. One way to prevent silly renames from Linux is to have proper close/remove semantics which was addressed up stream a few years ago. ------- Comment From ffilz.com 2010-03-25 14:02 EDT------- Ok, so it sounds like the way we want to proceed here is to have this patch included in RHEL 5.6: http://git.linux-nfs.org/?p=trondmy/nfs-2.6.git&a=commitdiff&h=a49c3c7736a2e77931dabc5bc4a83fb4b2da013e We understand silly renames will still occur in other situations, and that zOS will not be able to handle all silly rename variations. ------- Comment From ffilz.com 2010-04-06 16:00 EDT------- I have backported the following patch: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a49c3c7736a2e77931dabc5bc4a83fb4b2da013e In the process, to make the backport sane, I also backported this patch http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b39e625b6e75aa70e26c13f9378756bb5f2af032 Created attachment 404779 [details]
Backported patch to resolve issue
------- Comment (attachment only) From ffilz.com 2010-04-06 15:52 EDT-------
Thanks Frank, but the patch doesn't seem to apply cleanly to my RHEL5 tree. On what RHEL version did you base this? ------- Comment From ffilz.com 2010-04-13 16:57 EDT------- I had done the patch against RHEL 5.4, we have updated to RHEL 5.5 and I redid the patch. We will test it later today or tomorow. It looks like hunk1 for nfs4proc.c didn't apply because nfs4_call_async was changed to call rpc_new_task_wq instead of rpc_new_task. Created attachment 406346 [details]
Patch backported for RHEL 5.5
------- Comment (attachment only) From ffilz.com 2010-04-13 16:52 EDT-------
Oops, probably should have mentioned that I backported some of the same patches for 2.6.18 and put them into the test kernels on my people page: http://people.redhat.com/jlayton/ ...unless you see anything wrong with them, I'll probably go with those as they parallel the upstream fixes more closely. It looks like this also fixes an unrelated problem where the open_files list was being modified without holding the inode lock. ------- Comment From ffilz.com 2010-04-14 10:17 EDT------- Oh, thanks Jeff. Chris, could you verify Jeff's test kernel fixes your problem. ------- Comment From cvogan.com 2010-04-14 11:18 EDT------- The latest patched kernel (2.6.18-194.el560829) on a RH EL 5.5 system was successful. ------- Comment From ffilz.com 2010-04-14 11:34 EDT------- Great! Jeff, please let us know when your patch set is accepted into the stream for RHEL 5.6. Thanks everyone I've got a newer version of this patch in my test kernels that fixes a possible regression in the old patch (the close would have been queued to rpciod rather than nfsiod). It would be nice if IBM could test that patch too. The differences are minor, but I'd like to confirm that you don't see any problems from it. ------- Comment From ffilz.com 2010-05-03 14:00 EDT------- Jeff, your kernels here: have this update correct? That's correct. ------- Comment From cvogan.com 2010-05-04 09:49 EDT------- I applied kernel 2.6.18-197.el5.jtltest.101. I was unable to re-recreate the silly rename issue with a z/OS NFS server that was under heavy load. It looks like the test kernel from Jeff also resolves the issue. in kernel-2.6.18-207.el5 You can download this test kernel from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed. ------- Comment From cvogan.com 2010-07-19 23:38 EDT------- Hello I just tried to recreate the failing case with kernel: Linux reddy1 2.6.18-207.el5 #1 SMP Thu Jul 15 18:22:40 EDT 2010 i686 i686 i386 GNU/Linux My test completed successfully. I wrote a simple test case to test the failing condition to kernel: 2.6.18-194.el5PAE openclose.c #include<stdio.h> #include<stdlib.h> #include<fcntl.h> #include<string.h> #include<ctype.h> #include<time.h> #include<errno.h> int main(int argc, char *argv[]) { int fd=0; /*file descriptor*/ char startDir[256]; char mntpt[256]; char inputfile[256]; if (argc != 3) { printf("Usage: %s <mount point> <data file>\n", argv[0]); exit(1); } strcpy(mntpt, argv[1]); strcpy(inputfile, argv[2]); if (chdir(mntpt) < 0) { printf("Could not Change dir to mount point\n"); exit(1); } if ((fd=fopen( inputfile, "w+")) == 0) { printf("ERROR NO. %d\n",fd); printf("FAILED on open file %s\n",inputfile); exit(1); } if ((fclose(fd)) < 0 ) { printf("Could Not Close file %s\n",inputfile); exit(1); } printf("I just tried to remove file: %s\n",inputfile); if ((remove(inputfile)) < 0 ) { printf("Failed to remove file %s\n",inputfile); exit(1); } printf("I just tried to close file: %s\n",inputfile); } I just wrapped this C program around a looping shell script to run it many times in quick succession. *** Bug 439093 has been marked as a duplicate of this bug. *** test against comment #33 on RHEL5.6-Server-20101014.0 on i386 and x86_64, run it in loop and can completed successfully. Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Previously, Connectathon test cases performed on a z/OS NFSv4 server were regularly failing. While the file was being closed prior to the unlink call, the client did not wait for the close to complete before proceeding. This caused it to perform an inappropriate rename instead of unlinking the file, even though it was not required. With this update, removals on NFSv4 mounts should wait for outstanding close calls to complete before proceeding. This enhancement request was evaluated by the full Red Hat Enterprise Linux team for inclusion in a Red Hat Enterprise Linux minor release. As a result of this evaluation, Red Hat has tentatively approved inclusion of this feature in the next Red Hat Enterprise Linux Update minor release. While it is a goal to include this enhancement in the next minor release of Red Hat Enterprise Linux, the enhancement is not yet committed for inclusion in the next minor release pending the next phase of actual code integration and successful Red Hat and partner testing. ------- Comment From sglass.com 2010-11-30 15:40 EDT------- This feature has been verified. Thanks An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0017.html |