Bug 473396

Summary: NFS server reboot results in "Stale NFS file handle"
Product: [Fedora] Fedora Reporter: Dr. Tilmann Bubeck <tilmann>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 10CC: johnschmidt4, staubach, steved
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 496121 (view as bug list) Environment:
Last Closed: 2008-12-07 04:31:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 474449    
Attachments:
Description Flags
Patch to solve problem by changing order in /etc/init.d/nfs none

Description Dr. Tilmann Bubeck 2008-11-28 10:04:34 UTC
Description of problem:

I use FC10 (2.6.27.5-117.fc10.i686.PAE) as a NFS file server and different Fedora clients all behaving identical. Whenever I reboot the NFS server all clients get a "Stale NFS file handle" when accessing the NFS share.

I do use UDP (not TCP) for NFS and I do use "fsid" to ensure, that the filesystem ID is not changing. I'm using NFSv3.

I found out, that executing "exportfs -f" on the freshly rebootet file server does workaround the problem.

Using wireshark I found out, that:
 1. the fsid is indeed identical during reboots
    and matches the fsid written in /etc/exports
 2. I'm using UDP instead of TCP
 3. After reboot of the NFS server the client ("ls /home") asks for:
      GETATTR, FH: 0x83951db5 ... and gets a REPLY of NFS3ERR_STALE
      ACCESS,  FH: 0x83951db5 ... and gets a REPLY of NFS3ERR_STALE
 4. After "exportfs -f" the client asks for:
      ACCESS,  FH: 0x83951db5 ... and gets a REPLY of NFS3_OK
      GETATTR, FH: 0x83951db5 ... and gets a REPLY of NFS3_OK

This means, that the file handle is unchanged during the whole process. It is first valid, then the server reboots and it is stale, and after exportfs it is again valid.

Using Fedora 1 (2.4.31) it did not get this behaviour.

I build up and test server and I'm therefore able to reproduce the problem easily and test any fixes or ideas.

Does nyone knows what is wrong?

Version-Release number of selected component (if applicable):
kernel-2.6.27.5-117.fc10.i686.PAE
nfs-utils-1.1.4-1.fc10

How reproducible:
 always

Steps to Reproduce:
1. Export something via NFS
2. mount on client
3. reboot NFS server
4. watch "Stale NFS file handle"
5. execute "exportfs -f" on server
6. watch NFS working again.
  
Actual results:
Got "Stale NFS file handle"


Expected results:
No "Stale NFS file handle"


Additional info:

Comment 1 Dr. Tilmann Bubeck 2008-12-01 10:57:45 UTC
Created attachment 325213 [details]
Patch to solve problem by changing order in /etc/init.d/nfs

This patch fixes the problem by changing the start order in /etc/init.d/nfs.

Why? Before the patch the start order in /etc/init.d/nfs was:
  1. exportfs -r
  2. modprobe nfsd
     This also does (see modprobe.conf.dist): mount -t nfsd nfsd /proc/fs/nfsd

The patch changes the order to:
  1. modprobe nfsd 
     This also does (see modprobe.conf.dist): mount -t nfsd nfsd /proc/fs/nfsd
  2. exportfs -r

As you can read in "man exportfs" there are two modes: "legacy" and "new" depending on the existance of /proc/fs/nfsd. The unpatched version of /etc/init.d/nfs executes exportfs in lecacy mode and therefore feeds /var/lib/nfs/rmtab into the kernel. After "mount -t nfsd ..." the "new" mode is used. In this case the previously fed rmtab is disturbing because we have a mix of "legacy" and "new" mode. This results in a lot of problems as described in the initial post.

This mix could be fixed by issuing "exportfs -f" as a workaround. But this is not a good solution, because all NFS clients already had a "Stale NFS file handle". But they reconnect after "exportfs -f".

By applying the patch then "exportfs -r" will already find /proc/fs/nfsd and therefore use the "new" mode which is used throughout the rest of Fedora. Therefore no odd mix is created and everything goes smoothly.

Comment 2 Fedora Update System 2008-12-01 16:19:33 UTC
nfs-utils-1.1.4-2.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/nfs-utils-1.1.4-2.fc10

Comment 3 Fedora Update System 2008-12-01 16:23:51 UTC
nfs-utils-1.1.2-7.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/nfs-utils-1.1.2-7.fc9

Comment 4 Steve Dickson 2008-12-01 18:00:13 UTC
Till,

Thank you for your detailed analysis and Yes I think its accurate
to say the nfsd module should loaded before the exports
are created. 

If possible, could you give the 1.1.4-2 version of nfs-utils a try?
found at 
    http://koji.fedoraproject.org/koji/buildinfo?buildID=72546

If it works, I'll push it out to the rest of the community... tia..

Comment 5 Fedora Update System 2008-12-03 01:25:13 UTC
nfs-utils-1.1.4-2.fc10 has been pushed to the Fedora 10 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update nfs-utils'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2008-10642

Comment 6 Fedora Update System 2008-12-03 01:27:56 UTC
nfs-utils-1.1.2-7.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing-newkey update nfs-utils'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-10681

Comment 7 Dr. Tilmann Bubeck 2008-12-04 08:52:47 UTC
I tried nfs-utils-1.1.4-2.fc10.i386.rpm and it works correctly.
Thanks for the fast fix!

Comment 8 Fedora Update System 2008-12-07 04:26:11 UTC
nfs-utils-1.1.2-7.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2008-12-07 04:31:47 UTC
nfs-utils-1.1.4-2.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 10 johnschmidt4 2009-04-16 17:14:43 UTC
This is also present on RHEL 4.7 under these conditions.

root@a2a10 ~ rpm -qa | grep nfs
nfs-utils-1.0.6-87.EL4
nfs-utils-lib-1.0.6-8.z1
root@a2a10 ~ uname -a
Linux a2a10 2.6.9-78.0.8.ELsmp #1 SMP Wed Nov 5 07:10:44 EST 2008 i686 i686 i386 GNU/Linux
root@a2a10 ~ cat /etc/redhat-release 
Red Hat Enterprise Linux ES release 4 (Nahant Update 7)

Comment 11 Peter Staubach 2009-04-16 18:18:39 UTC
Would you mind creating a bugzilla for RHEL-4 for this, please?