LTC 267 We would like this added to the 7.1 errata. Thanks! Hardware Environment: Software Environment: RH 7.1, but believe this to be present on any linux release running NFS components. Steps to Reproduce: 1. Exporting a directory such as /lib, the client can successfully use files in this directory. If, though, there exists another link for the same file in /lib in a different directory, we can accidentally find an improper dentry when doing sanity checks in nfs's fh_verify(). We are still investigating this and getting closer to finding the real bug. Again, directory structures must be as follows: /lib/libxxx.so ----> inode 1234 /tmp/lib/libxxx.so ----> also inode 1234 Now, export /lib. All will be well with the client until something happens to touch the /tmp/lib file on the server. The client can now be told it has a stale file handle as it looks up and finds the dentry for /tmp/lib, which does not compare favorably with the export of /lib. More details on this as we discover them. I have been working this with Dave Wilder. ------- Additional Comments From Kyle Grieser(yuf.com) 2001-09-18 17:22 ------- Earlier today, we found the section of nfsd code that looks up the dentry and returns the wrong entry. In nfsfh.c, in the routine find_fh_dentry(), there is the following code: result = nfsd_iget(sb, ino, generation); if (IS_ERR(result) || (!(result->d_flags & DCACHE_NFSD_DISCONNECTED ) || (!S_ISDIR(result->d_inode->i_mode) && ! needpath)) { up(&sb->s_nfsd_free_path_sem); err = PTR_ERR(result); if (IS_ERR(result)) goto err_out; if ((result->d_flags & DCACHE_NFSD_DISCONNECTED)) nfsdstats.fh_anon++; return result; } We have found that while "needpath" is indeed one, as it shouold be to be sure we check the parentage of the dentry, we are still returning the result here instead of continuing with the more brute force code to do parent checking. This is due to the dentry not having the DCACHE_NFSD_DISCONNECTED flag set. I have speculated that this would not be set if the dentry was created by the dcache.c code (generic, non nfs code). Due to this, we are tricked into thinking that it is okay to skip parent checking (since it is "connected"). In the case of files with more than 1 link, this is not okay. Since the base vfs layer created the dentry, and not NFS, it cannot assume that DCACHE_NFSD_DISCONNECTED work has been done. Given the above, the check was changed to the following, making the code do parent checking if the file has more than a single link (and therefore more than one dentry, which could reside outside the exported filesystem): if (IS_ERR(result) || (!(result->d_flags & DCACHE_NFSD_DISCONNECTED) && (result->d_inode->i_nlink == 1)) || (!S_ISDIR(result->d_inode->i_mode) && ! needpath)) { up(&sb->s_nfsd_free_path_sem); err = PTR_ERR(result); if (IS_ERR(result)) goto err_out; if ((result->d_flags & DCACHE_NFSD_DISCONNECTED)) nfsdstats.fh_anon++; return result; } Test runs have confirmed that this makes the problem go away, but we still have some loose ends to follow due to some strange behaviour of the customers client system. We believe these to be unrelated, or possibly caused by a bad network card or cable. We have not seen a stale file handle problem since implementing this fix. Also, since this code is the hackiest, nastiest, ugliest I've seen in some time, it is possible we may have subtley affected behaviour for other scenarios. I believe it should not adversely affect anything though, since it would simply make us do extra work where we may have avoided it. My belief is that any file with more than 1 link, if it is on a filesystem that is not fully exported, must have these checks done, since the extra links could reside outside the exported portion of the filesystem. After some more testing, and potentially some testing by the customer, we will draft a note to the linux kernel list and see if the linux community concurs with this simple fix. Google searches reveal that there are others that have suffered from this, so this may be a welcomed fix. Of course, you never know :-). -yuf ------- Additional Comments From Kyle Grieser(yuf.com) 2001-09-19 15:03 ------- Some correspondence with the code maintainer complicates this issue. It seems they are rewriting much of the nfs code, as well as parts of the vfs layer, and other filesystems that can be exported. This supposedly may fix this bug, but the current "patches" apply to the v2.4.10pre6 version of the kernel. In other words, this doesn't help anyone that is running kernels that are currently out there. The patches do not apply easily to any previous versions, and so are not a good solution for our customer. We are attempting to verify if the code rewrites fix the problem, which gives the customer something to do when they get to v2.4.10. Until then, however, we would like to recommend that they simply continue using the workaround that we gave them last week. Removing the existance of extra links to an inode that lie outside the exported hierarchy alleviates the problem. While our patch is simple, and also avoids the problem, we doubt it will be accepted for the distribution. For us to maintain this patch for all versions of all linux OS's until v2.4.10 is very impractical. ------- Additional Comments From Kyle Grieser(yuf.com) 2001-09-21 13:06 ------- We have verified that the patches from the NFS maintainer do indeed fix the problem. Unfortunately they apply to kernel version 2.4.9. At this time we would like to recommend to the customer that they implement the workaround to avoid the problem until such a time that they move to the fixed version of code. We are not sure when this will be. Perhaps an update to v2.4.9, but maybe later. Issuing our patch would be an administrative headache, and does have a couple minor side effects. Dave has come up with a repeat-by that can be used to test to see if the problem has been fixed. He will update this bug report with that information soon. ------- Additional Comments From David Wilder(wilder.com) 2001-09-21 13:18 ------- This is the test procedure that can be used to verify if NFS server bug still exists in a future version of the kernel. According to the code maintainer the fix will not be included until some version later than 2.4.10. Equipment requirements: Two linux systems one NFS server one NFS client. The version of the software on the client is not important. Initial server setup: 1. Create a test directory. Insure that no portion of the path to the directory has been exported. ( the parent is not listed in /etc/exports) # cd /home # mkdir test 2. Create a file in the test directory. # touch /home/test/file 3. Create a hard link to the file in its parent directory. # ln /home/test/file /home/file 4. Add the test directory to the file /etc/exports added to file /etc/exports /home/test *(rw,no_root_squash) Once the setup is complete follow these steps to test for the problem. Note: the order that the steps are performed is very important. If you make mistake you must re-boot and start the test over. It is important that very little activity is occurring on the server during the test. 1. re-boot the server (this is an important step, don't skip it) # init 6 2. On the server logon as root and start nfs (unless nfs is configures to auto-start thus it is already running). # /etc/rc.d/init.d/nfs start 3. On the server access the two files (the order of the access is vital) # cat /home/test/file # cat /home/file 4. On the client mount the exported test directory. # mount -t nfs server-hostname:/home/test /mnt 5. On the client access the test file. # cat /mnt/file If the problem exists the client will display the following error message. "cat: /mnt/file: Stale NFS file handle" On the server the following messages will be displayed on the console (and logged into /etc/log/messages) "nfsd Security: //file bad export" If no messages are printed on ether system the problem did not occur.