Bug 1418130

Forgot to say, manual mounts using the same options do work.

nomad

I'll take a look, thanks for the report.

Can you post /etc/amd.conf please?

[global]
auto_dir =			/atm
# Fedora doesn't support NFSv2, use the amd NFSv3 server.
auto_nfs_version =              3
browsable_dirs =		no
dismount_interval =		300
fully_qualified_hosts =		no
log_file =			/var/log/amd
log_options =			fatal,noinfo,nostats,debug
map_type =			file
nfs_retransmit_counter =	20
nfs_retry_interval =		16
normalize_hostnames =		no
plock =				yes
print_pid =			no
print_version =			no
restart_mounts =		yes
search_path =			/etc/amd
selectors_on_default =		yes

[/g]
map_name = amd.group
map_type = file

[/n]
map_name = amd.network
map_type = file

[/users]
map_name = amd.homes
map_type = file

(In reply to nomad from comment #4)
> [global]
> auto_dir =			/atm
> # Fedora doesn't support NFSv2, use the amd NFSv3 server.
> auto_nfs_version =              3

Right, I had to check that you had auto_nfs_version set.

I can't duplicate this on CentOS, think it's 7.3.

How about a debug log please.

I included the entire contents of /var/log/amd after setting log_options to include debug. Are there additional settings I can/should use?

re-posted here for convenience:

: [89] ; cat /var/log/amd
Jan 31 16:04:36 quillpen amd[31054]/debug: dir name /vol/attic_h
Jan 31 16:04:36 quillpen amd[31054]/error: '/atm/chicken-b/vol/attic_h': mount: Invalid argument
Jan 31 16:04:36 quillpen amd[31034]/warn:  find_nfs_srvr: NFS mount failed, trying again with NFSv2/UDP
Jan 31 16:04:36 quillpen amd[31055]/error: '/atm/chicken-b/vol/attic_h': mount: Protocol not supported
Jan 31 16:04:36 quillpen amd[31034]/error: /n/chicken-b/attic_h: mount (amfs_cont): Protocol not supported

/etc/redhat-release says 7.3.1611.

nomad

(In reply to nomad from comment #7)
> /etc/redhat-release says 7.3.1611.

I'm pretty sure that's what I'm testing with too.
Can you post the full debug log please.

(In reply to nomad from comment #6)
> I included the entire contents of /var/log/amd after setting log_options to
> include debug. Are there additional settings I can/should use?
> 
> re-posted here for convenience:
> 
> : [89] ; cat /var/log/amd
> Jan 31 16:04:36 quillpen amd[31054]/debug: dir name /vol/attic_h
> Jan 31 16:04:36 quillpen amd[31054]/error: '/atm/chicken-b/vol/attic_h':
> mount: Invalid argument
> Jan 31 16:04:36 quillpen amd[31034]/warn:  find_nfs_srvr: NFS mount failed,
> trying again with NFSv2/UDP

Oh right, the "Protocol not supported" message is probably coming from
the fall back to NFSv2 so that's sensible and sounds correct.

Which means we need to ficus on the "Invalid argument" error.

It's probably best to send the log output to a specific log file and
set all the debug options so we don't miss anything.

What I use (which gets very verbose output) to make sure I don't
miss anything is:

log_file = /var/log/amd
log_options = all
debug_options = all

Created attachment 1250900 [details]
log output with debug fully enabled

The host was a bit busy but I did manage to capture the attempted mount of /n/chicken-b/data.

(In reply to nomad from comment #10)
> Created attachment 1250900 [details]
> log output with debug fully enabled
> 
> The host was a bit busy but I did manage to capture the attempted mount of
> /n/chicken-b/data.

Thanks for the log.

According to the log the mount(2) system call is returning EINVAL
with parameters that look ok.

But I see it's trying NFSv4, getting an error and falling back to
NFSv2 and, of course, failing because v2 isn't supported by the
RHEL kernel.

I think it should be falling back to v3 not v2, I'll need to have
a look and see if I can work out why it's doing that.

Should v4 mounts work for this host?
What happens if you specify NFSv3 in the mount options?

Oh, and what kernel revision are you using?
Output of "uname -r" will give us that.

Umm, sorry, could you try and get me a debug log of this without
noatime and ro options (IOW a successful mount attempt)?

> ...kernel revision...
3.10.0-514.2.2.el7.x86_64

> Should v4 mounts work for this host?

I would think so. This is a standard CentOS 7 host mounting a filesystem exported by a NetApp filer. When I try to manually do the mount it works:
 
: || lvd@quillpen / [114] ; sudo mount.nfs -o ro,noatime chicken-b:/vol/data /media
: || lvd@quillpen / [115] ; df -h !$
df -h /media
Filesystem           Size  Used Avail Use% Mounted on
chicken-b:/vol/data  3.7T  2.1T  1.7T  55% /media

mount(8) shows:

chicken-b:/vol/data on /media type nfs4 (ro,noatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.232.18,local_lock=none,addr=192.168.232.31)


> What happens if you specify NFSv3 in the mount options?

Interesting. amd.conf has the following setting:

auto_nfs_version =              3

And I don't think there's anything in amd.network that would override that. However, every mount seems to be nfs4.

That said, forcing it with opts:=vers=3 on the type:=nfs line seems to work.

chicken-b/attic_h	type:=nfs;opts:=vers=3,noatime,ro,hard,proto=tcp;rhost:=chicken-b;rfs:=/vol/attic_h

mount(8) shows:

chicken-b:/vol/attic_h on /atm/chicken-b/vol/attic_h type nfs (ro,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=2049,timeo=16,retrans=20,sec=sys,local_lock=none,addr=192.168.232.31)

(I'd swear I tried that early on and it didn't work. Must have been a different combination of things.)

> a debug log of this without noatime and ro options

I'll upload a new log which shows both the successful v3 mount and the successful noatime,ro mount.

Created attachment 1256200 [details]
log file showing successful mounts

(In reply to nomad from comment #14)
> > ...kernel revision...
> 3.10.0-514.2.2.el7.x86_64

Thanks, I'll have a look at the NFS mount options evaluation
in that and see if I can spot anything.

> 
> > Should v4 mounts work for this host?
> 
> I would think so. This is a standard CentOS 7 host mounting a filesystem
> exported by a NetApp filer. When I try to manually do the mount it works:
>  
> : || lvd@quillpen / [114] ; sudo mount.nfs -o ro,noatime chicken-b:/vol/data
> /media
> : || lvd@quillpen / [115] ; df -h !$
> df -h /media
> Filesystem           Size  Used Avail Use% Mounted on
> chicken-b:/vol/data  3.7T  2.1T  1.7T  55% /media

Right, but I don't know what mount.nfs(8) might be doing to
the options before calling mount(2) and we know the mount(2)
call in amd is returning EINVAL. From memory I thought
mount.nfs(8) didn't do much at all to the options for NFSv4
mounts ....

I couldn't see where this happens looking at recent kernel
code either, it's not that straight forward to work it out.

> 
> mount(8) shows:
> 
> chicken-b:/vol/data on /media type nfs4
> (ro,noatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,
> port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.232.18,local_lock=none,
> addr=192.168.232.31)
> 
> 
> > What happens if you specify NFSv3 in the mount options?
> 
> Interesting. amd.conf has the following setting:
> 
> auto_nfs_version =              3

LOL, that option is for the amd internal NFS server which
handles the triggering of mounts.

Historically amd used NFSv2 for this but Fedora and RHEL-7
kernels don't enable NFSv2 any more so the internal server
needed to be updated to use NFSv3.

It doesn't actually affect options used for mounts that are
made based on map entries.

> 
> And I don't think there's anything in amd.network that would override that.
> However, every mount seems to be nfs4.
> 
> That said, forcing it with opts:=vers=3 on the type:=nfs line seems to work.
> 
> chicken-b/attic_h
> type:=nfs;opts:=vers=3,noatime,ro,hard,proto=tcp;rhost:=chicken-b;rfs:=/vol/
> attic_h
> 
> mount(8) shows:
> 
> chicken-b:/vol/attic_h on /atm/chicken-b/vol/attic_h type nfs
> (ro,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,
> port=2049,timeo=16,retrans=20,sec=sys,local_lock=none,addr=192.168.232.31)

The thing is that, for NFSv4 mounts, amd checks the return and only
tries NFSv3 for two possible error returns and neither are EINVAL.

So there's no fallback to NFSv3 made and it then tries an NFSv2
mount which will always fail because the kernel is configured to
not support v2.

I'm not sure yet if I should just add EINVAL to that, I'll need to
look at it again. There's also the question of why we are getting
EINVAL which needs to be understood.

Unfortunately I have some time critical work (and have been working
on that for a while now) so I need to delay this.

But I think you have a workaround for the time being so I hope that
won't be too much of a problem.

Ian