Bug 264661 - NFS Permission Denied Errors
Summary: NFS Permission Denied Errors
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: nfs-utils
Version: 7
Hardware: All
OS: All
medium
low
Target Milestone: ---
Assignee: Steve Dickson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 692968
TreeView+ depends on / blocked
 
Reported: 2007-08-29 18:36 UTC by Jeroen van Meeuwen
Modified: 2011-04-01 20:18 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 692968 (view as bug list)
Environment:
Last Closed: 2007-09-04 09:35:52 UTC
Type: ---


Attachments (Terms of Use)
tshark capture of a couple of mount attempts (14.36 KB, application/x-bzip)
2007-08-30 19:07 UTC, Jeroen van Meeuwen
no flags Details
network trace mounting pinky /home nfs share from elwood (19.40 KB, application/octet-stream)
2007-09-01 13:41 UTC, Jeroen van Meeuwen
no flags Details

Description Jeroen van Meeuwen 2007-08-29 18:36:08 UTC
Description of problem:
Shares configured on FC6 and F7 give permission errors from FC6 and F7

Version-Release number of selected component (if applicable):
(F7 x86) rpm -qv nfs-utils
nfs-utils-1.1.0-3.fc7
(F7 x86_64) rpm -qv nfs-utils
nfs-utils-1.0.12-4.fc7
(FC6 x86) rpm -qv nfs-utils
nfs-utils-1.0.10-14.fc6

How reproducible:

Steps to Reproduce:
1. Export /home and /data on FC6 or F7
2. Export /data on F7
3. Mount any of these from FC6 or F7
  
Actual results:
[root@elwood ~]# mount -vvvv -t nfs -o tcp pinky:/home /mnt
mount: trying 10.10.10.50 prog 100003 vers 3 prot tcp port 2049
mount: trying 10.10.10.50 prog 100005 vers 3 prot tcp port 798
mount: pinky:/home failed, reason given by server: Permission denied

Expected results:
Mounted NFS share

Additional info:
THIS IS the FC6 i386 box
[root@pinky ~]# uname -r
2.6.20-1.2962.fc6
[root@pinky ~]# cat /etc/exports
/home *.kanarip.com(rw,sync,no_root_squash)
/data *.kanarip.com(rw,sync,no_root_squash)
[root@pinky ~]# exportfs -ra
[root@pinky ~]# exportfs
/home           *.kanarip.com
/data           *.kanarip.com
[root@pinky ~]# cat /proc/fs/nfs/exports 
# Version 1.1
# Path Client(Flags) # IPs
/home   *.kanarip.com(rw,no_root_squash,sync,wdelay,no_subtree_check)
[root@pinky ~]# 

THIS IS the F7 i386 box
[root@elwood ~]# uname -a
Linux elwood.kanarip.com 2.6.22.1-41.fc7 #1 SMP Fri Jul 27 18:10:34 EDT 2007
i686 i686 i386 GNU/Linux
[root@elwood ~]# cat /etc/exports
/data           *.kanarip.com(rw,wdelay,no_root_squash,no_subtree_check)
[root@elwood ~]# exportfs
/data           *.kanarip.com
[root@elwood ~]# cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/data   *.kanarip.com(rw,no_root_squash,sync,wdelay,no_subtree_check)
[root@elwood ~]# 

MOUNT FROM FC6 box:
[root@pinky ~]# mount -t nfs -vvvv -o tcp elwood:/data /mnt
mount: trying 10.10.10.58 prog 100003 vers 3 prot tcp port 2049
mount: trying 10.10.10.58 prog 100005 vers 3 prot tcp port 50510
mount: elwood:/data failed, reason given by server: Permission denied
[root@pinky ~]# 

MOUNT either FROM F7 i386 / x86_64:
[root@slevin ~]# uname -a
Linux slevin.kanarip.com 2.6.22.1-41.fc7 #1 SMP Fri Jul 27 18:21:43 EDT 2007
x86_64 x86_64 x86_64 GNU/Linux
[root@slevin ~]# mount -t nfs -vvvv -o tcp elwood:/data /mnt
mount: trying 10.10.10.58 prog 100003 vers 3 prot tcp port 2049
mount: trying 10.10.10.58 prog 100005 vers 3 prot tcp port 50510
mount: elwood:/data failed, reason given by server: Permission denied
[root@slevin ~]# mount -t nfs -vvvv -o tcp pinky:/home /mnt
mount: trying 10.10.10.50 prog 100003 vers 3 prot tcp port 2049
mount: trying 10.10.10.50 prog 100005 vers 3 prot tcp port 798
mount: pinky:/home failed, reason given by server: Permission denied
[root@slevin ~]# 

DNS resolving works (on all ends):
[root@slevin ~]# host pinky
pinky.kanarip.com has address 10.10.10.50
[root@slevin ~]# host 10.10.10.50
50.10.10.10.in-addr.arpa domain name pointer pinky.kanarip.com.
[root@slevin ~]# host elwood
elwood.kanarip.com has address 10.10.10.58
[root@slevin ~]# host 10.10.10.58
58.10.10.10.in-addr.arpa domain name pointer elwood.kanarip.com.

Allowing world access doesn't help.
Firewalls are turned off.
SELinux are turned off.
Shared directories have other read and execute permissions set.
rpcinfo -p <host> and showmount -e <host> show good results.

Edit the first line of /etc/exports which shares /data and /home, and allow
world to use home:
[root@pinky ~]# cat /etc/exports 
/home *(rw,sync,no_root_squash)
/data *.kanarip.com(rw,sync,no_root_squash)
[root@pinky ~]# exportfs -ra
[root@pinky ~]# exportfs
/data           *.kanarip.com
/home           <world>
[root@pinky ~]# cat /proc/fs/nfs/exports 
# Version 1.1
# Path Client(Flags) # IPs
/home   *,*.kanarip.com(rw,no_root_squash,sync,wdelay,no_subtree_check)
[root@pinky ~]#

Comment 1 Jeroen van Meeuwen 2007-08-30 19:07:21 UTC
Created attachment 181621 [details]
tshark capture of a couple of mount attempts

Attaching capture of a couple of mount attempts

Comment 2 Steve Dickson 2007-08-31 13:24:48 UTC
Just as I suspected, the server is denying the mount due to
(I'm guessing) your exports. 

Fire up wiresharke and read in the data file you posted.
Then use 'rpc.program == 100005' as a display filter 
and you will see for your self, that the server is 
returning an "ERR_ACCESS"

Comment 3 Jeroen van Meeuwen 2007-09-01 11:53:22 UTC
Upgraded elwood to rawhide:
[root@elwood ~]# rpcinfo -p pinky
rpcinfo: can't contact portmapper: RPC: Unknown host

while other machines can get rpcinfo:
[root@vito ~]# rpcinfo -p pinky
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
[...truncated...]

And get rpcinfo from the rawhide box as well:

[root@vito ~]# rpcinfo -p elwood
   program vers proto   port
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
[...truncated...]

In fact, FC6 is now able to mount the rawhide machine's share, but still no go
vice-versa, or from any other FC6 machine

Comment 4 Steve Dickson 2007-09-01 12:01:19 UTC
hmm... this is a strange one... On elwood (the rawhide machine) 
can you post a network trace of that failure? It may
shed some light on whats going on.... 

In general, a "Unknown Host" means its a DNS problem but
I'm assuming other apps are not having the same problem..

Question: is IPv6 enabled on any of your network interfaces?



Comment 5 Jeroen van Meeuwen 2007-09-01 13:41:28 UTC
Created attachment 184481 [details]
network trace mounting pinky /home nfs share from elwood

Attached network trace; please note that the valid NFS traffic you see is a
mount from pinky (10.10.10.50) on elwood's(10.10.10.58) /data/backup share

Still gets ERR_ACCESS on:
[root@pinky ~]# cat /etc/exports 
/home *.kanarip.com(rw,insecure,sync,no_root_squash)

and:

[root@elwood ~]# rpcinfo -p pinky
rpcinfo: can't contact portmapper: RPC: Unknown host
[root@elwood ~]# rpcinfo -p localhost
rpcinfo: can't contact portmapper: RPC: Unknown host
[root@elwood ~]# rpcinfo -p 127.0.0.1
rpcinfo: can't contact portmapper: RPC: Unknown host

Comment 6 Jeroen van Meeuwen 2007-09-01 13:43:38 UTC
Creating attachment doesn't set back to assigned ;-)

Additional info for elwood (rawhide, the one with the rpcinfo problems):
[root@elwood ~]# rpm -qv rpcbind
rpcbind-0.1.4-7.fc8


Comment 7 Jeroen van Meeuwen 2007-09-01 13:52:13 UTC
One more thing, these are both FC6 machines, one that is working as an NFS
server (vito), and one that isn't (pinky):

[root@vito ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
[root@vito ~]# uname -a
Linux vito.kanarip.com 2.6.19-1.2895.fc6 #1 SMP Wed Jan 10 19:28:18 EST 2007
i686 i686 i386 GNU/Linux
[root@vito ~]# cat /etc/fedora-release
Fedora Core release 6 (Zod)
[root@vito ~]# rpm -qv nfs-utils
nfs-utils-1.0.10-14.fc6
[root@vito ~]# 


[root@pinky ~]# mount
/dev/hda3 on / type ext3 (rw)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /sys type sysfs (rw)
/dev/md0 on /home type ext3 (rw,usrquota,grpquota)
elwood.kanarip.com:/data/backup on /backup type nfs (rw,tcp,addr=10.10.10.58)
[root@pinky ~]# uname -a
Linux pinky.kanarip.com 2.6.20-1.2962.fc6 #1 SMP Tue Jun 19 19:27:14 EDT 2007
i686 i686 i386 GNU/Linux
[root@pinky ~]# cat /etc/fedora-release 
Fedora Core release 6 (Zod)
[root@pinky ~]# rpm -qv nfs-utils
nfs-utils-1.0.10-14.fc6
[root@pinky ~]# 


Comment 8 Jeroen van Meeuwen 2007-09-02 18:06:35 UTC
Manually mounting:

nfsd on /proc/fs/nfsd type nfsd (rw)

solved the problem. Any reason why this mount failed?


Comment 9 Jeroen van Meeuwen 2007-09-04 09:35:52 UTC
OK, definitely solved (I can work again ;p), but unsure why nfsd isn't mounted
(haven't tested reboot) Will reopen if necessary. Thanks Steve!

Comment 10 Steve Dickson 2007-09-04 11:10:01 UTC
WOW... Take a look at your /etc/modprobe.d/modprobe.conf.dist file
and make sure the following lines exists:

install nfsd /sbin/modprobe --first-time --ignore-install nfsd && \
 { /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }

install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc && \
 { /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs > /dev/null 2>&1 ||:; }

Both  /proc/fs/nfsd and /var/lib/nfs/rpc_pipefs should be mounted
during modprobe time....



Comment 11 Lars Kellogg-Stedman 2011-04-01 20:13:45 UTC
We've just run into this problem on two of our RHEL5.5 systems.  That is, systems that had previously been exporting NFS filesystems without a problem suddenly started reporting "permission denied" despite entries like this in /etc/exports:

  /mnt/somefilesystem *(ro)

The solution was to manually mount /proc/fs/nfsd.  

/etc/modprobe.d/modprobe.conf.dist is unmodified from the stock configuration and includes the necessary install/remove rules:

  install nfsd /sbin/modprobe --first-time --ignore-install nfsd && { /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }
  remove nfsd { /bin/umount /proc/fs/nfsd > /dev/null 2>&1 || :; } ; /sbin/modprobe -r --first-time --ignore-remove nfsd

This happened on unrelated systems, maintained by two distinct groups of people.

Since the mount operation is idempotent (if the filesystem is already mounted, running "mount -t nfsd nfsd /proc/fs/nfsd" simply reports an error), we've simply made this part of the NFS startup script.  It seems that this might be a good idea in general.


Note You need to log in before you can comment on or make changes to this bug.