Bug 638507

Summary: Cannot set up NFS server on F14 beta system
Product: [Fedora] Fedora Reporter: He Rui <rhe>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 14CC: andy, fabrice, jlayton, schorschi, steved
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: nfs-utils-1.2.3-0.fc14 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-11 23:10:47 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
Server side nfs mount attempt packet capture none

Description He Rui 2010-09-29 04:04:07 EDT
Description of problem:
Tried to set up nfs server on F14 beta system but failed by both manual and nfs configuration GUI.

For manually, I:
0. Open the firewall and selinux
1. configured /etc/exports with: /media *(rw),
2. service nfs start
3. service rpc* start

On another machine, 'showmount -e NFS SERVER IP' displayed nfs directory correctly, but when mount, it waited for a long time and then said 'timeout'   

Version-Release number of selected component (if applicable):
nfs-utils-1.2.2-6.fc14.x86_64
nfs-utils-lib-1.1.5-2.fc14.x86_64

How reproducible:
always


Additional info:
I've been setting up nfs server for many times on previous fedora systems using the same way.
Comment 1 Fabrice Bellet 2010-10-05 15:09:29 EDT
I confirm the problem. Here is some more debug information :

NFS: nfs_fhget(0:15/2 ct=1)
NFS: nfs_update_inode(0:15/2 ct=2 info=0x27e7f)
NFS: permission(0:15/2), mask=0x1, res=0
NFS: atomic_lookup(0:15/2), mnt
NFS: lookup(/mnt)
NFS: nfs_fhget(0:15/44990481 ct=1)
--> nfs_follow_mountpoint()
nfs_follow_mountpoint: enter
--> nfs_do_submount()
nfs_do_submount: submounting on /mnt
--> nfs4_xdev_get_sb()
<-- nfs4_xdev_get_sb() = -116 [error]
nfs_do_submount: done
<-- nfs_do_submount() = ffffffffffffff8c
nfs_follow_mountpoint: done, returned -116
<-- nfs_follow_mountpoint() = -116
NFS: dentry_delete(/mnt, 0)
Comment 2 Andrew Haveland-Robinson 2010-10-05 20:41:41 EDT
Not sure if this is related, but since upgrading 2 machines from fc13 to fc14 beta, the previously working NFS shares now won't mount.
selinux disabled, iptables stopped.

showmount -e 192.168.0.3
/Archives/MySQL_Backups 192.168.0.0/255.255.255.0

mount -v -t nfs -o noatime,nodiratime,intr,rw,bg,nosuid,nfsvers=3 192.168.0.3:/Archives/MySQL_Backups /Archives/MySQL_Backups
mount.nfs: trying text-based options 'intr,bg,nfsvers=3,addr=192.168.0.3'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.0.3 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.0.3 prog 100005 vers 3 prot UDP port 46835
mount.nfs: mount(2): Stale NFS file handle
mount.nfs: backgrounding "192.168.0.3:/"
mount.nfs: mount options: "intr,bg,nfsvers=3,addr=192.168.0.3"

netstat -anp --tcp --udp | grep rpc (edited)
tcp 0.0.0.0:51546  6775/rpc.mountd
tcp 0.0.0.0:34500  1162/rpc.statd
tcp 0.0.0.0:875    6759/rpc.rquotad
tcp 0.0.0.0:111    6562/rpcbind
tcp :::59039       1162/rpc.statd
tcp :::111         6562/rpcbind
udp 0.0.0.0:801    6562/rpcbind
udp 0.0.0.0:875    6759/rpc.rquotad
udp 0.0.0.0:914    1162/rpc.statd
udp 0.0.0.0:46835  6775/rpc.mountd
udp 0.0.0.0:44795  1162/rpc.statd
udp 0.0.0.0:111    6562/rpcbind
udp :::801         6562/rpcbind
udp :::35907       1162/rpc.statd
udp :::111         6562/rpcbind

server: /var/log/messages
mountd[31036]: authenticated mount request from 192.168.0.8:997 for /Archives/MySQL_Backups (/Archives/MySQL_Backups)

client:
mount to NFS server '192.168.0.3' failed: Stale NFS file handle, retrying

On both machines
kernel-2.6.35.4-28.fc14.x86_64
nfs-utils-1.2.2-6.fc14.x86_64
nfs-utils-lib-1.1.5-2.fc14.x86_64

mount -t nfs -o noatime,nodiratime,intr,rw,bg,nosuid 192.168.0.3:/Archives/MySQL_Backups /Archives/MySQL_Backups

Always getting stale handles even though the server appears to authenticate the request, and it's a showstopper for me.
Comment 3 Fabrice Bellet 2010-10-06 09:34:14 EDT
Weird : running the nfs server with selinux in enforcing mode seems to make it work. With selinux disabled or in permissive mode, it fails.
Comment 4 Steve Dickson 2010-10-06 09:37:18 EDT
Can some one post a bzip2 binary network trace of this
failure... Something similar to:
    tshark -w /tmp/data.pcap host <server>
    bzip2 /tmp/data.pcap
Comment 5 Fabrice Bellet 2010-10-06 09:40:54 EDT
...but it works for me with the latest koji build of nfs-utils : http://koji.fedoraproject.org/koji/buildinfo?buildID=198843
Comment 6 Andrew Haveland-Robinson 2010-10-06 23:10:37 EDT
Created attachment 452030 [details]
Server side nfs mount attempt packet capture
Comment 7 Andrew Haveland-Robinson 2010-10-06 23:17:13 EDT
Fabrice, curious that it seems that nfs expects selinux to be present, but many people will need it to work without selinux too. I'll have a look at koji's build.

Steve, thanks for the info - I installed wireshark, and ran this on the server after stopping other services getting in the way:

tshark -w /tmp/data.pcap host 192.168.0.3

and ran this on 192.168.0.8 just afterwards:
mount -t nfs -o noatime,nodiratime,intr,rw,bg,nosuid 192.168.0.3:/Archives/MySQL_Backups /Archives/MySQL_Backups

I hope the trace will help.
Comment 8 Andrew Haveland-Robinson 2010-10-06 23:50:40 EDT
Fabrice, thanks for the link, it works! :-))))
(I think it'll fix He Rui's problem too).

For others' benefit, until reaching the repo, just download from:
http://koji.fedoraproject.org/koji/buildinfo?buildID=198843
and run on the server:
rpm -Uvh nfs-utils-1.2.3-0.fc14.x86_64.rpm
/etc/init.d/nfs restart

I first installed it on the client, no mount.
then installed on the server, and it mounted.

(Also, this message didn't appear on the console as with the previous:
mount.nfs: backgrounding "192.168.0.3:/Archives/MySQL_Backups"
mount.nfs: mount options: "intr,bg,addr=192.168.0.3")

I also ran the mount on another unpatched machine with success, so it appears to be just a server-side problem.

Steve, the packet trace is academic now if fixed, but it still may be useful.

Regards, Andy :-)
Comment 9 Fedora Update System 2010-10-07 11:04:33 EDT
nfs-utils-1.2.3-0.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/nfs-utils-1.2.3-0.fc14
Comment 10 Fedora Update System 2010-10-07 15:53:50 EDT
nfs-utils-1.2.3-0.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update nfs-utils'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/nfs-utils-1.2.3-0.fc14
Comment 11 Steve Dickson 2010-10-08 10:53:18 EDT
Guys, would you mind give the Fedora update push some good Karma? 
As soon as that happens I'll push it into stable...
Comment 12 He Rui 2010-10-11 05:28:34 EDT
(In reply to comment #11)
> Guys, would you mind give the Fedora update push some good Karma? 
> As soon as that happens I'll push it into stable...

Just added a third positive karma, it should be pushed to stable. Thanks.
Comment 13 Fedora Update System 2010-10-11 23:10:43 EDT
nfs-utils-1.2.3-0.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 14 Schoschi Decker 2010-11-28 18:06:26 EST
Not sure if this related, but also did Fedora 13 to 14 upgrade and once working NFS server configuration now fails, mountd refuses to start.  Reinstalled Fedora 14 from official media and still see same issue, mountd refuses to start.  No warnings or errors in messages log.