Bug 218994

Summary: Permission denied when trying to mount a nfs share
Product: [Fedora] Fedora Reporter: Dominik 'Rathann' Mierzejewski <dominik>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED RAWHIDE QA Contact: Ben Levenson <benl>
Severity: low Docs Contact:
Priority: low    
Version: 6CC: gilboad, jan.kratochvil
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-06-29 09:39:17 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Dominik 'Rathann' Mierzejewski 2006-12-08 17:54:43 EST
Description of problem:
Mounting a NFS share between an i686 laptop and x86_64 workstation fails with
permission denied. The same laptop can mount NFS shares from other machines just
fine (I tried serving NFS from another FC5 i686 workstation, and a FC4 i686 server).

Version-Release number of selected component (if applicable):
nfs-utils-1.0.10-4.fc6

How reproducible:
Always

Steps to Reproduce:
1. ryvius:# cat /etc/exports 
/home/media             192.168.2.19(rw,async)
2. neeya:# mount.nfs ryvius:/home/media /mnt/media -v
mount: trying 192.168.2.15 prog 100003 vers 3 prot tcp port 2049
mount: trying 192.168.2.15 prog 100005 vers 3 prot udp port 676
mount: ryvius:/home/media failed, reason given by server: Permission denied
  
Actual results:
Mount fails.

Expected results:
Mount should succeed.

Additional info:

# showmount -e ryvius
Export list for ryvius:
/home/media 192.168.2.19

There are no unusual log entries on the server:
Dec  8 23:52:29 ryvius mountd[32235]: authenticated mount request from
neeya.pekin.waw.pl:922 for /home/media (/home/media)

One oddity I noticed - running rpc.mountd under strace reveals:
open("/proc/fs/nfsd/filehandle", O_RDWR) = -1 ENOENT (No such file or directory)
open("/proc/fs/nfs/filehandle", O_RDWR) = -1 ENOENT (No such file or directory)
On another FC4 i686 machine that acts as a NFS server, the file
/proc/fs/nfsd/filehandle exists.

I have no more clues.
Comment 1 Dominik 'Rathann' Mierzejewski 2006-12-08 17:56:21 EST
Forgot to add: neither the client nor the server have selinux or iptables enabled.
Comment 2 Dominik 'Rathann' Mierzejewski 2006-12-08 18:06:03 EST
Looks like shutting down all NFS services, modprobe -r nfs nfsd and restarting
those services caused it to work again, but I wonder what happened there.
Comment 3 Steve Dickson 2006-12-09 07:52:57 EST
This looks like it could be a mount bug that is fixed in
the updated nfs-utils package. So please 'yum update nfs-utils' 
to the nfs-utils-1.0.10-4.fc6 package... 

Comment 4 Dominik 'Rathann' Mierzejewski 2006-12-12 07:59:35 EST
(In reply to comment #3)
> This looks like it could be a mount bug that is fixed in
> the updated nfs-utils package. So please 'yum update nfs-utils' 
> to the nfs-utils-1.0.10-4.fc6 package... 

You'll notice that in comment #1 I wrote that I was using the updated
1.0.10-4.fc6 package.
Comment 5 Steve Dickson 2006-12-14 08:00:02 EST
It sounds like something happen to your exports on the
server sided... If this happens again... please try
'exportfs -arv' to see if it clear up the problem...
Comment 6 Jan Kratochvil 2007-03-17 11:16:43 EDT
Confirming I saw the same behavior now, running on RawHide:
kernel-xen-2.6.19-1.2898.2.3.fc7.i686
nfs-utils-1.0.12-3.fc7.i386

No restarts of anything in userland helped.  Just later RMMOD of everything and
a simple `/etc/init.d/nfs restart' (START would be probably enough) got it working.
No EXPORTFS was run to get it working.  No reproducibility guide, though.
Comment 7 Steve Dickson 2007-03-19 13:11:27 EDT
There is no failure or warning messages in /var/log/messages
that might give a clue as to what is happeing?

Comment 8 Jan Kratochvil 2007-03-19 13:15:41 EDT
There really wasn't any at least in the case of mine.
Also I believe it could get reassiged to `kernel' at this point although it
won't help much in its resolving.
Comment 9 Dominik 'Rathann' Mierzejewski 2007-03-19 14:11:02 EDT
There was nothing in my case. I haven't been able to reproduce it ever since.
Comment 10 Steve Dickson 2007-03-19 14:54:26 EDT
OK.. 

Could you please post a bzip2 binary tethereal network trace
of of the failing mount? Something similar to 
    tethereal -w /tmp/bz218994.pcap host <server>

Also is Selinux in the picture?

Comment 11 Jan Kratochvil 2007-03-19 15:22:19 EDT
(In reply to comment #10)
> Could you please post a bzip2 binary tethereal network trace
> of of the failing mount? Something similar to 
>     tethereal -w /tmp/bz218994.pcap host <server>

I did not store such dump but I was checking it that time by ethereal and there
was nothing interesting.  There was a mount request for / and some (do not
remember the error code) access denied sent back by the (broken) server. 
Despite the server logger `authenticated mount request from ... for / (/)'.

> Also is Selinux in the picture?

No SELinux on my machine (SELINUX=disabled).
Comment 12 Steve Dickson 2007-03-21 13:16:49 EDT
Are the clients multi-homed? Could it be possible that
the nfs request is coming from an IP address that is
not being exported? 

I'm noting that the exported filesystem is only exported 
to one client... does using the '*' wild-card (i.e. *(rw,async) )
make any difference?   
Comment 13 Jan Kratochvil 2007-03-21 13:22:09 EDT
Client was a XEN guest with single IP in my case.
I was trying also the '*' export there.
I was also using `exportfs -rav' and `exportfs -ravf'.
Comment 14 Dominik 'Rathann' Mierzejewski 2007-03-23 15:34:35 EDT
(In reply to comment #12)
> Are the clients multi-homed? Could it be possible that
> the nfs request is coming from an IP address that is
> not being exported?

Not in my case. Single IP for the client and server.

> I'm noting that the exported filesystem is only exported 
> to one client... does using the '*' wild-card (i.e. *(rw,async) )
> make any difference?   

I still cannot reproduce it again. I'll post back if anything happens.
Comment 15 Gilboa Davara 2007-06-17 16:37:23 EDT
... I think I'm hitting the same problem since the last update.
I cannot mount share from other machines; I can't mount my own shares from
localhost.
- No errors, /var/log/messages is clear. (Apart from the normal "authenticated
mount request from...")
- I can only see the EACCESS using wireshark.
- hosts.allow/deny are empty. 
- Putting wild-card in /etc/exports doesn't solve the problem.

$ rpm -qa | grep nfs-utils
nfs-utils-1.0.10-12.fc6.x86_64
nfs-utils-lib-1.0.8-7.2.x86_64

Any idea?
Comment 16 Steve Dickson 2007-06-18 07:38:15 EDT
WRT Comment #15.... Please post the output of mount -v which should tell you
why the mount is failing also you need to run wireshare as root 
which should take care of the EACCESS problem... 
Comment 17 Gilboa Davara 2007-06-18 08:13:55 EDT
$ mount -v 192.168.1.2:/mnt/Work /tmp/mnt
mount: no type was given - I'll assume nfs because of the colon
mount: trying 192.168.1.2 prog 100003 vers 3 prot tcp port 2049
mount: trying 192.168.1.2 prog 100005 vers 3 prot udp port 2050
mount: 192.168.1.2:/mnt/Work failed, reason given by server: Permission denied
$ exportfs | grep Wrok
/mnt/Work       192.168.1.0/255.255.255.0
$ ifconfig eth0 | grep "inet addr"
inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
$ tethereal -i lo
  ...
  0.003840  192.168.1.2 -> 192.168.1.2  Portmap V2 GETPORT Call MOUNT(100005)
V:3 UDP
  0.003869  192.168.1.2 -> 192.168.1.2  Portmap V2 GETPORT Reply (Call In 21)
Port:2050
  0.003914  192.168.1.2 -> 192.168.1.2  MOUNT V3 NULL Call
  0.003987  192.168.1.2 -> 192.168.1.2  MOUNT V3 NULL Reply (Call In 23)
  0.004047  192.168.1.2 -> 192.168.1.2  MOUNT V3 MNT Call /mnt/Work
  0.052859  192.168.1.2 -> 192.168.1.2  MOUNT V3 MNT Reply (Call In 25)
Error:ERR_ACCESS
  ...
Comment 18 Dominik 'Rathann' Mierzejewski 2007-06-19 03:37:17 EDT
Have you tried my workaround from Comment #2?
Comment 19 Gilboa Davara 2007-06-19 04:33:41 EDT
Dominik,

I couldn't rmmod nfsd - even though all rpc/nfs-related services are down, the
module usage count is still > 0.
After a reboot, I'm still getting ERR_ACCESS.

- Gilboa

Comment 20 Dominik 'Rathann' Mierzejewski 2007-06-19 14:37:22 EDT
Gilboa,

then it might be a different bug. Try running rpc.mountd on the server under
strace and see if that gives any more clues.

Do the files /proc/fs/nfsd/filehandle and /proc/fs/nfs/filehandle exist on the
server?
Comment 21 Steve Dickson 2007-06-19 14:43:03 EDT
> 0.052859  192.168.1.2 -> 192.168.1.2  MOUNT V3 MNT Reply (Call In 25)
Error:ERR_ACCESS
hmm... its mountd denying the mount... 

Please added the '-d all' argument to mountd
(via /etc/sysconfig/nfs) and redstart mountd.
Then please post the debugging output which
should be in /var/log/messages.
Comment 22 Gilboa Davara 2007-06-24 06:10:18 EDT
Steve and Dominik,

I've finally had time to install F7 on the problematic machine and it solved the
problem.
Once I have some time to, I'll pull the FC6 image from my backups and do some
stracing on mountd.


- Gilboa
Comment 23 Steve Dickson 2007-06-29 09:39:17 EDT
Thank you for your time and effort... at this point I'm going to
close this bug, but please reopen it if the problem does 
reappear...