Bug 652786

what does is the output of 'ps ax | grep nfsd' after the service is stopped?

(In reply to comment #1)
> what does is the output of 'ps ax | grep nfsd' after the service is stopped?

Here is the result of stopping the nfs service:

[gus@falcon ~]$ sudo service nfs stop
Shutting down NFS mountd:                                  [  OK  ]
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Shutting down NFS services:                                [  OK  ]
[gus@falcon ~]$ ps ax|grep nfs
28923 pts/7    S+     0:00 grep --color=auto nfs
[gus@falcon ~]$

As you can see, if the nfs service is stopped from the command line before the system is shut down, it stops cleanly. When the system is shut down with nfs running, I never see the "Shutting down ..." messages. It's as if the init script never gets called.

I did all my testing from run level 3 to eliminate GUI problems, and I don't have any clients connected. Because the system is shutting down, I'm not sure how to capture the console messages. I could try to set up a serial console and capture the shutdown sequence.

I also turned off almost all other services. I'll see what the bare minimum is to trigger the problem.

I found the bug. It exists because the subsystem lock file name does not match the init script name. The /etc/rc.d/rc script checks to see if a subsytem is running by using the name of the init script for the subsystem file. In the case of the NFS subsystem, the NFS init script creates a subsysetm file /var/lock/subsys/nfsd. It should be nfs (no d on the end).

I noticed that this subsystem check affects several other systems also, for example CUPS and rsyslog. I wrote a tiny little program to check the subsystem status (derived from /etc/rc.d/rc):

#!/bin/bash
# check if a subsystem is running based on what needs to be killed
runlevel=6

# First, run the KILL scripts.
for i in /etc/rc$runlevel.d/K* ; do

	# Check if the subsystem is already up.
	subsys=${i#/etc/rc$runlevel.d/K??}
	echo "Checking for $subsys"
	[ -f /var/lock/subsys/$subsys ] || [ -f /var/lock/subsys/$subsys.init ] || echo "$subsys does not exist"

done



Patch for /etc/init.d/nfs:

--- nfs.orig    2010-11-24 10:24:15.191119091 -0800
+++ nfs 2010-11-24 13:26:52.369559897 -0800
@@ -124,7 +124,7 @@
        RETVAL=$?
        echo
        touch /var/lock/subsys/rpc.mountd
-       touch /var/lock/subsys/nfsd
+       touch /var/lock/subsys/nfs
 
        # Let rpc.idmapd know that rpc.mountd just started
        [ -x /usr/sbin/rpc.idmapd ] && /sbin/service rpcidmapd condstart

Created attachment 462775 [details]
patch to fix nfs shutdown problem

I missed a few places. The attached patch gets all the remaining nfs vs nfsd substitutions.

*** Bug 655726 has been marked as a duplicate of this bug. ***

nfs-utils-1.2.3-2.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/nfs-utils-1.2.3-2.fc14

nfs-utils-1.2.3-2.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update nfs-utils'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/nfs-utils-1.2.3-2.fc14

There is a similar problem in rpc.idmapd which is also part of nfs-utils. Simple patch attached to fix.

Created attachment 464390 [details]
patch to fix rpcidmapd shutdown problem

nfs-utils-1.2.3-2.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

This affects nfs-utils-1.2.2-6.fc13 in Fedora 13 too.

Can an update for Fedora 13 please be pushed as well? This bug often prevents a server from being remotely rebootable, as the failure of NFS to come down cleanly during the reboot process causes files to be left open and the system locks up trying to unmount them.