Bug 1387694

Summary: Document rpcbind not mandatory with NFSv4-only server
Product: Red Hat Enterprise Linux 7 Reporter: Marko Myllynen <myllynen>
Component: doc-Storage_Administration_GuideAssignee: Marek Suchánek <msuchane>
Status: CLOSED CURRENTRELEASE QA Contact: Yongcheng Yang <yoyang>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.3CC: bfields, harshula, jiyin, jlayton, kzhang, mlinden, msuchane, myllynen, rhel-docs, steved, swhiteho, xzhou, yoyang
Target Milestone: rcKeywords: Documentation
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-19 12:18:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1470091    

Description Marko Myllynen 2016-10-21 15:07:18 UTC 
Document URL:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/s1-nfs-start.html

Section Number and Name:
8.6. Starting and Stopping NFS

Describe the issue: 
The section states:

To run an NFS server, the rpcbind[1] service must be running.

However, this is not needed if the server is configured NFSv4 only.

Suggestions for improvement:
Reword the above and also explain where fitting how to run NFSv4 only server.

Additional information:
Comment 19 J. Bruce Fields 2017-03-15 13:22:17 UTC 
So, for testing, what will be needed is to
- perform the configuration steps recommended in the documentation
- check that basic NFSv4 functionality still works
- check that NFSv3 mounts fail as expected.
- check that only port 2049 is open (rpcbind and statd should not be running, rpc.mountd should be running but have no open ports).
Comment 24 J. Bruce Fields 2017-12-21 16:41:49 UTC 
I checked out a RHEL7 machine and experimented a bit.  Here's some draft documentation:

By default, an RHEL7 server supports NFSv2, NFSv3, and NFSv4 (including minor versions 4.0, 4.1, and 4.2).

To configure a server to support only NFSv4 (including all its minorversions), add a line to /etc/sysconfig/nfs:

  RPCNFSDARGS="-N 2 -N 3 -U"

The change will take effect the next time the nfs server is started or restarted, after which clients attempting to mount with NFSv2 or NFSv3 will get an error like "requested NFS version or transport protocol is not supported".

The server will still listen for RPCBIND, MOUNT, and NSM protocol calls, although these are not necessary in the NFSv4-only case.  To turn off those as well, edit /etc/sysconfig/nfs to add:

  RPCMOUNTDOPTS="-N 2 -N 3"

Then run:

  systemctl mask rpc-statd.service rpcbind.service rpcbind.socket

The change will take affect next time the NFS server is started or restarted.  After this, clients attempting to mount over NFSv2 or NFSv3 will hang.  Also, the server itself will be unable to mount NFSv2 or NFSv3 filesystems, since mounts also depend on rpc.statd.

Notes:

- I did some quick tests, but somebody else should check and make sure this works for them.
- after this netstat -l shows nfs as the only listening nfs service.
- I'm not sure why NFSv2/3 mounts are hanging or if that's fixable.
- The rpc.nfsd manpage claims that -U is the default, but experiments show otherwise.  In any case, -U will become unnecessary in RHEL8, which will no longer support NFSv4 over UDP.
- I thought masking the rpc-bind service would be enough, but it turned out I needed to mask the socket too.  I may just not understand how socket-activated services are meant to work.
Comment 25 Marko Myllynen 2018-01-04 15:31:21 UTC 
Thanks for looking into this. I tested can we push this even a bit further in case of gssproxy/idmapd is not in use, this seems to work:

# cat /etc/sysconfig/nfs
RPCNFSDARGS="-N 2 -N 3 -U"
RPCMOUNTDOPTS="-N 2 -N 3 -p 875"
GSS_USE_PROXY="no"
# systemctl mask rpc-statd.service rpcbind.service rpcbind.socket rpcidmapd.service rpc-gssd.service rpcgssd.service nfs-idmapd.service nfs-idmap.service gssproxy.service auth-rpcgss-module.service
# systemctl start nfs-server

However, here we're probably pushing a bit too far for something that could be considered to be a generic / supported solution, we see:

Process: 5470 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=1/FAILURE)

(although this is harmless) but in general perhaps gssproxy/idmapd are recommended to be enabled.

With this procedure I see no other additional processes except for rpc.mountd be running after starting nfs-server.service, masking nfs-mountd.service will cause mounts to fail on RHEL 6 clients with "mount.nfs4: access denied by server while mounting ..." which is perhaps a bit surprising given that it should not be needed in the NFSv4-only case.

So this is just for reference, not sure what would be the minimal but still supportable configuration.

Thanks.
Comment 26 Yongcheng Yang 2018-01-05 09:15:17 UTC 
(In reply to Marko Myllynen from comment #25)

> With this procedure I see no other additional processes except for
> rpc.mountd be running after starting nfs-server.service, masking
> nfs-mountd.service will cause mounts to fail on RHEL 6 clients with
> "mount.nfs4: access denied by server while mounting ..." which is perhaps a
> bit surprising given that it should not be needed in the NFSv4-only case.

Just reading this from chaper "Network File System (NFS)" from both rhel6 and rhle7 "Storage Administration Guide":
'''
As such, NFSv4 does not need to interact with rpcbind, lockd, and rpc.statd daemons. The rpc.mountd daemon is required on the NFS server to set up the exports, but is not involved in any over-the-wire operations.
'''

So IMO the rpc.mountd is essential for the nfs server, maybe we can emphasize it in the man page rpc.mountd(8).
Comment 27 Marko Myllynen 2018-01-16 09:27:41 UTC 
(In reply to Yongcheng Yang from comment #26)
> (In reply to Marko Myllynen from comment #25)
> 
> > With this procedure I see no other additional processes except for
> > rpc.mountd be running after starting nfs-server.service, masking
> > nfs-mountd.service will cause mounts to fail on RHEL 6 clients with
> > "mount.nfs4: access denied by server while mounting ..." which is perhaps a
> > bit surprising given that it should not be needed in the NFSv4-only case.
> 
> Just reading this from chaper "Network File System (NFS)" from both rhel6
> and rhle7 "Storage Administration Guide":
> '''
> As such, NFSv4 does not need to interact with rpcbind, lockd, and rpc.statd
> daemons. The rpc.mountd daemon is required on the NFS server to set up the
> exports, but is not involved in any over-the-wire operations.
> '''

Bruce, I realize this might be nearing a point where the discussion would be more appropriate for an upstream development project but can you please confirm that rpc.mountd is a hard-requirement for NFSv4 and there are no plans to change that?

Thanks.
Comment 28 J. Bruce Fields 2018-01-29 14:46:20 UTC 
(In reply to Marko Myllynen from comment #27)
> Bruce, I realize this might be nearing a point where the discussion would be
> more appropriate for an upstream development project but can you please
> confirm that rpc.mountd is a hard-requirement for NFSv4 and there are no
> plans to change that?

There are currently no plans to change that.

Note that rpc.mountd should not be listening on the network when run with -N 2 -N 3.  If it is, we should file a bug and get that fixed.  (And people can firewall it off if that helps.)

We definitely need all those gss-related services, and we need rpc.idmapd.

Note that none of those should be listening on the network either.

Our assumption is that people are concerned about having more running services listening on open ports, not about just having more processes running.  The NFS server will probably always depend on some userland daemons for help, as it's inconvenient to do things like kerberos handshakes or name lookups inside the kernel.
Comment 29 mlinden 2018-02-01 12:59:42 UTC 
So it seems that rpc.mountd is listening on the network.
I will create a new bz to see if we can stop it from listening when -N2 -N 3 is set.

# netstat -nap |grep rpc.mount
tcp        0      0 0.0.0.0:20048           0.0.0.0:*               LISTEN      1510/rpc.mountd     
tcp6       0      0 :::20048                :::*                    LISTEN      1510/rpc.mountd     
udp        0      0 0.0.0.0:20048           0.0.0.0:*                           1510/rpc.mountd     
udp6       0      0 :::20048                :::*                                1510/rpc.mountd     
unix  2      [ ]         DGRAM                    29840    1510/rpc.mountd      

Perhaps a note should be added to firewall off those ports if they are not needed is sufficient for now.
Comment 30 mlinden 2018-02-01 13:08:14 UTC 
Bug about rpc.mountd created https://bugzilla.redhat.com/show_bug.cgi?id=1540985
Comment 31 mlinden 2018-02-01 13:19:26 UTC 
Ignore my 2 previous comments.

I DONT see rpc.mountd listening to any ports.
Comment 32 Steve Whitehouse 2018-03-28 09:16:09 UTC 

Is there anything eles required from the NFS team at this stage in order to progress this bug?
Comment 33 Marek Suchánek 2018-03-28 17:26:50 UTC 
I've edited Bruce's draft and prepared a new section in the Storage Administration Guide. See the built preview:

http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-Red_Hat_Enterprise_Linux-7-Storage_Administration_Guide-branch-NFSv4-only-1387694/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#nfs4-only

The "Starting and Stopping the NFS Server" section, which has been reformatted, now links to the NFSv4-only procedure in Prerequisites:

http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-Red_Hat_Enterprise_Linux-7-Storage_Administration_Guide-branch-NFSv4-only-1387694/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#s1-nfs-start

Bruce, can you please review the new section and let me know if it's OK? Also, could you please provide sample output of `netstat -l` that shows that nfs is the only listening nfs service?

Thanks,

Marek
Comment 34 J. Bruce Fields 2018-04-05 19:35:51 UTC 
(In reply to Marek Suchanek from comment #33)
> Bruce, can you please review the new section and let me know if it's OK?

I saw one problem (my fault): the "mask" isn't enough to stop rpcbind immediately on its own.  Adding "--now" loosk like an easy fix; so make that command:

  systemctl mask --now rpc-statd.service rpcbind.service rpcbind.socket

> Also, could you please provide sample output of `netstat -l` that shows that
> nfs is the only listening nfs service?

On a RHEL7 machine before the described changes:

# netstat -ltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:nfs             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:36069           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:52364           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:sunrpc          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:mountd          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN     
tcp        0      0 localhost:smtp          0.0.0.0:*               LISTEN     
tcp6       0      0 [::]:34941              [::]:*                  LISTEN     
tcp6       0      0 [::]:nfs                [::]:*                  LISTEN     
tcp6       0      0 [::]:sunrpc             [::]:*                  LISTEN     
tcp6       0      0 [::]:mountd             [::]:*                  LISTEN     
tcp6       0      0 [::]:12432              [::]:*                  LISTEN     
tcp6       0      0 [::]:56881              [::]:*                  LISTEN     
tcp6       0      0 [::]:12434              [::]:*                  LISTEN     
tcp6       0      0 localhost:7092          [::]:*                  LISTEN     
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
udp        0      0 localhost:323           0.0.0.0:*                          
udp        0      0 0.0.0.0:37190           0.0.0.0:*                          
udp        0      0 0.0.0.0:876             0.0.0.0:*                          
udp        0      0 localhost:877           0.0.0.0:*                          
udp        0      0 0.0.0.0:mountd          0.0.0.0:*                          
udp        0      0 0.0.0.0:38588           0.0.0.0:*                          
udp        0      0 0.0.0.0:nfs             0.0.0.0:*                          
udp        0      0 0.0.0.0:bootpc          0.0.0.0:*                          
udp        0      0 0.0.0.0:sunrpc          0.0.0.0:*                          
udp6       0      0 localhost:323           [::]:*                             
udp6       0      0 [::]:57683              [::]:*                             
udp6       0      0 [::]:876                [::]:*                             
udp6       0      0 [::]:mountd             [::]:*                             
udp6       0      0 [::]:40874              [::]:*                             
udp6       0      0 [::]:nfs                [::]:*                             
udp6       0      0 [::]:sunrpc             [::]:*                             

After:

# netstat -ltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:nfs             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN     
tcp        0      0 localhost:smtp          0.0.0.0:*               LISTEN     
tcp6       0      0 [::]:nfs                [::]:*                  LISTEN     
tcp6       0      0 [::]:12432              [::]:*                  LISTEN     
tcp6       0      0 [::]:12434              [::]:*                  LISTEN     
tcp6       0      0 localhost:7092          [::]:*                  LISTEN     
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
udp        0      0 localhost:323           0.0.0.0:*                          
udp        0      0 0.0.0.0:bootpc          0.0.0.0:*                          
udp6       0      0 localhost:323           [::]:*
Comment 35 Marek Suchánek 2018-04-06 13:32:35 UTC 
Bruce, thanks for the review and the additional information. I've included it in the new section.

Kexin, can you please assign somebody from QA to review the new documentation? It's a procedure to configure the NFS server to work only with NFSv4 and to disable rpcbind.

See section 8.7.7. "Configuring an NFSv4-only Server" in this built preview:

https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/nfs-serverconfig#nfs4-only

Thanks!
Comment 36 Marko Myllynen 2018-04-11 07:20:28 UTC 
The documentation looks good to me but perhaps we could consider one more addition (sorry for not bringing up this earlier):

RPCMOUNTDOPTS="-N 2 -N 3 -u"

This would be consistent with RPCNFSDARGS and will explicitly disable advertising UDP for mounting which is inappropriate with NFSv4 (see e.g. https://bugzilla.redhat.com/show_bug.cgi?id=606260).

Thanks.
Comment 37 Yongcheng Yang 2018-04-11 08:24:49 UTC 
(In reply to Marek Suchanek from comment #35)

I'll take this one and test the procedure a bit.

> Kexin, can you please assign somebody from QA to review the new
> documentation? It's a procedure to configure the NFS server to work only
> with NFSv4 and to disable rpcbind.
> 
> See section 8.7.7. "Configuring an NFSv4-only Server" in this built preview:
> 
> https://doc-stage.usersys.redhat.com/documentation/en-us/
> red_hat_enterprise_linux/7/html/storage_administration_guide/nfs-
> serverconfig#nfs4-only
> 
> Thanks!
Comment 38 J. Bruce Fields 2018-04-11 15:57:38 UTC 
(In reply to Marko Myllynen from comment #36)
> The documentation looks good to me but perhaps we could consider one more
> addition (sorry for not bringing up this earlier):
> 
> RPCMOUNTDOPTS="-N 2 -N 3 -u"
> 
> This would be consistent with RPCNFSDARGS and will explicitly disable
> advertising UDP for mounting which is inappropriate with NFSv4 (see e.g.
> https://bugzilla.redhat.com/show_bug.cgi?id=606260).

NFSv4 clients don't use the mount protocol, so with RPCMOUNTDOPTS="-N 2 -N3", rpc.mountd should not be listening on the network at all.  The "-u" is therefore unnecessary.
Comment 39 Marko Myllynen 2018-04-12 07:29:29 UTC 
(In reply to J. Bruce Fields from comment #38)
> (In reply to Marko Myllynen from comment #36)
> > The documentation looks good to me but perhaps we could consider one more
> > addition (sorry for not bringing up this earlier):
> > 
> > RPCMOUNTDOPTS="-N 2 -N 3 -u"
> > 
> > This would be consistent with RPCNFSDARGS and will explicitly disable
> > advertising UDP for mounting which is inappropriate with NFSv4 (see e.g.
> > https://bugzilla.redhat.com/show_bug.cgi?id=606260).
> 
> NFSv4 clients don't use the mount protocol, so with RPCMOUNTDOPTS="-N 2
> -N3", rpc.mountd should not be listening on the network at all.  The "-u" is
> therefore unnecessary.

Correct, thanks for pointing this out.
Comment 40 Yongcheng Yang 2018-04-19 06:08:14 UTC 
(In reply to Yongcheng Yang from comment #37)
> (In reply to Marek Suchanek from comment #35)
> 
> I'll take this one and test the procedure a bit.
> 
> > Kexin, can you please assign somebody from QA to review the new
> > documentation? It's a procedure to configure the NFS server to work only
> > with NFSv4 and to disable rpcbind.
> > 
> > See section 8.7.7. "Configuring an NFSv4-only Server" in this built preview:
> > 
> > https://doc-stage.usersys.redhat.com/documentation/en-us/
> > red_hat_enterprise_linux/7/html/storage_administration_guide/nfs-
> > serverconfig#nfs4-only
> > 
> > Thanks!

I have checked that the "Configuring NFSv4-only Server" procedure is correct on both rhel6 and rhel7. And have already generated an new test case to cover it.

Looks like in rhel6 attempting to mount NFSv3 will hang a bit longer (2 minutes) as comment #24 mentioned. But it's not big deal IMO.

Moving to VERIFIED (maybe we can close it) now.
Comment 41 Marek Suchánek 2018-04-19 12:18:19 UTC 
Yongcheng, thanks very much for the review and for creating the test case.

I'm closing this bug as CURRENTRELEASE. The published content can be found here:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/nfs-serverconfig#nfs4-only