Bug 972363

Summary: nfs-secure.service does not start automatically even though enabled
Product: [Fedora] Fedora Reporter: Dean Hunter <deanhunter>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 19CC: antokarag, bfields, bojan, Colin.Simpson, igeorgex, jlayton, mkosek, rcritten, steved
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: nfs-utils-1.2.8-3.0.fc19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-18 19:57:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
patch: systemd: enable nfs-secure.service by default and fix description in nfs.target none

Description Dean Hunter 2013-06-08 22:07:24 UTC 
Description of problem:

nfs-secure.service (aka rpcgssd.service?) does not start automatically even though enabled:

[root@fedora19 ~]# systemctl enable nfs-secure.service
[root@fedora19 ~]# reboot
  ...
[root@fedora19 ~]# systemctl status nfs-secure.service
nfs-secure.service - Secure NFS
   Loaded: loaded (/usr/lib/systemd/system/nfs-secure.service; enabled)
   Active: inactive (dead)
  ...

This causes a failure when referencing a directory auto-mounted with FreeIPA:

Jun  8 16:43:38 fedora19 kernel: [   47.916801] FS-Cache: Loaded
Jun  8 16:43:38 fedora19 kernel: [   47.921886] Key type dns_resolver registered
Jun  8 16:43:38 fedora19 kernel: [   47.936967] RPC: Registered named UNIX socket transport module.
Jun  8 16:43:38 fedora19 kernel: [   47.936971] RPC: Registered udp transport module.
Jun  8 16:43:38 fedora19 kernel: [   47.936972] RPC: Registered tcp transport module.
Jun  8 16:43:38 fedora19 kernel: [   47.936973] RPC: Registered tcp NFSv4.1 backchannel transport module.
Jun  8 16:43:38 fedora19 kernel: [   47.960075] FS-Cache: Netfs 'nfs' registered for caching
Jun  8 16:43:38 fedora19 kernel: [   47.992355] NFS: Registering the id_resolver key type
Jun  8 16:43:38 fedora19 kernel: [   47.992372] Key type id_resolver registered
Jun  8 16:43:38 fedora19 kernel: [   47.992373] Key type id_legacy registered
Jun  8 16:43:53 fedora19 kernel: [   63.006062] RPC: AUTH_GSS upcall timed out.
Jun  8 16:43:53 fedora19 kernel: [   63.006062] Please check user daemon is running.

Using systemctl to start nfs-secure.service resolves the problem.


Version-Release number of selected component (if applicable):

nfs-utils-1.2.8-2.0.fc19.x86_64


How reproducible: consistent
Comment 1 Dean Hunter 2013-06-10 16:47:07 UTC 
This problem does not occur in Fedora 18 with nfs-utils-1.2.7-6.fc18.x86_64.
Comment 2 Dean Hunter 2013-07-11 16:36:58 UTC 
Hello?

(I don't think there is anybody in there.)
Comment 3 Dean Hunter 2013-07-11 19:59:27 UTC 
I think I know what happened.  Some got a little zealous correcting the NFS systemd unit files:

From nfs-utils-1.2.7-6.fc18.x86_64:

/usr/lib/systemd/system/nfs-secure.service
[Unit]
Description=Secure NFS
Requires=var-lib-nfs-rpc_pipefs.mount
After=syslog.target var-lib-nfs-rpc_pipefs.mount

[Service]
Type=forking
StandardError=syslog+console
EnvironmentFile=-/etc/sysconfig/nfs
ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/nfs-secure-server.service
[Unit]
Description=Secure NFS Server
Requires=var-lib-nfs-rpc_pipefs.mount nfs-server.service
After=syslog.target var-lib-nfs-rpc_pipefs.mount nfs-server.service 

[Service]
Type=forking
StandardError=syslog+console
EnvironmentFile=-/etc/sysconfig/nfs
ExecStart=/usr/sbin/rpc.svcgssd $RPCSVCGSSDARGS

[Install]
WantedBy=multi-user.target

The nfs-secure-server.service should be wanted by nfs.target; ie. the nfs-server wants all its parts.  However, nfs-secure.service, being of the client side is NOT wanted by nfs.target.  But someone made a mistake and changed BOTH service files.

From nfs-utils-1.2.8-2.0.fc19.x86_64:

/usr/lib/systemd/system/nfs-secure.service
[Unit]
Description=Secure NFS
Requires=var-lib-nfs-rpc_pipefs.mount
After=syslog.target var-lib-nfs-rpc_pipefs.mount

[Service]
Type=forking
StandardError=syslog+console
EnvironmentFile=-/etc/sysconfig/nfs
ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS

[Install]
WantedBy=nfs.target

/usr/lib/systemd/system/nfs-secure-server.service
[Unit]
Description=Secure NFS Server
Requires=var-lib-nfs-rpc_pipefs.mount nfs-server.service
After=syslog.target var-lib-nfs-rpc_pipefs.mount nfs-server.service 

[Service]
Type=forking
StandardError=syslog+console
EnvironmentFile=-/etc/sysconfig/nfs
ExecStart=/usr/sbin/rpc.svcgssd $RPCSVCGSSDARGS

[Install]
WantedBy=nfs.target

As a work-around, I have added this to the FreeIPA client install script I use:

sed -i s/WantedBy=nfs.target/WantedBy=multi-user.target/ \
  /usr/lib/systemd/system/nfs-secure.service
Comment 4 Dean Hunter 2013-07-12 01:08:01 UTC 
Oops, the workaround should be:

  cat /usr/lib/systemd/system/nfs-secure.service \
    | sed -e s/WantedBy=nfs.target/WantedBy=multi-user.target/ \
    > /etc/systemd/system/nfs-secure.service
Comment 5 Martin Kosek 2013-07-22 06:47:14 UTC 
Note that this bug is related to Bug 970595.

I filed an upstream ticket for FreeIPA to enable nfs.target by default. If I understand this problem correctly, it should fix it:

https://fedorahosted.org/freeipa/ticket/3807
Comment 6 Dean Hunter 2013-07-22 12:05:42 UTC 
Martin,

nfs.target will start all the NFS server processes.  For ipa-client-automount I think you only want to start the NFS client processes.
Comment 7 Dean Hunter 2013-07-22 14:26:49 UTC 
Here is what I have found necessary to make NFS work the way I think it should:

For an NFS server:

  systemctl enable nfs.target

For an NFS server with Kerberos add:

  systemctl enable nfs-secure-server.service

For an NFS client with Kerberos:

  cat /usr/lib/systemd/system/nfs-secure.service \
    | sed -e s/WantedBy=nfs.target/WantedBy=multi-user.target/ \
    > /etc/systemd/system/nfs-secure.service

  systemctl enable nfs-secure.service
Comment 8 Fedora Update System 2013-07-30 11:46:59 UTC 
nfs-utils-1.2.8-3.0.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/FEDORA-2013-13591/nfs-utils-1.2.8-3.0.fc19
Comment 9 Fedora Update System 2013-08-02 03:50:20 UTC 
nfs-utils-1.2.8-3.0.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 10 Dean Hunter 2013-08-08 20:25:12 UTC 
I understand what you did to nfs.target, but I am not sure how you think that solves this problem.  I need an NFS client that will use Kerberos.  Now, instead of not starting nfs-secure.service, all of the NFS server suite is started along with nfs-secure.service.
Comment 11 JM 2013-08-15 15:26:10 UTC 
Same problem here rpc.gssd (nfs-secure.service) is dead after a reboot (nfs-secure.service is enabled):

-----
systemctl status nfs-secure.service
nfs-secure.service - Secure NFS
   Loaded: loaded (/usr/lib/systemd/system/nfs-secure.service; enabled)
   Active: inactive (dead)
-----

-----
RPC: AUTH_GSS upcall timed out.
Please check user daemon is running.
-----

I use nfs-utils-1.2.8-3.0.fc19.x86_64 and it looks this version has the same problem with nfs-secure.service.
Comment 12 JM 2013-08-15 16:15:55 UTC 
The patch from comment #4 (replace Wanted=nfs.target with WantedBy=multi-user.target) solved the problem. 

Of course to enable nfs.target starts rpc.gssd as well but also all NFS-Server programs, for a client this is not necessary so the patch from comment #4 is much better.
Comment 13 Bojan Smojver 2013-08-17 04:20:33 UTC 
Of course, as a workaround, one can also stick a symlink to nfs-secure.service into /usr/lib/systemd/system/multi-user.target.wants. But, yeah, annoying. Causes my autofs mounted NFS file systems to hang nautilus for about 15 seconds.
Comment 14 Jeff Layton 2013-08-17 10:57:24 UTC 
Created attachment 787532 [details]
patch: systemd: enable nfs-secure.service by default and fix description in nfs.target

I agree. The current layout of systemd targets and services is really odd. The nfs.target says:

    Description=Network File System Server

...but rpc.gssd is a _client_ side daemon. Looking at that file though, it's seems like the description is just wrong. It should probably read:

    Description=Network File System Client and Server Support

...since that target just seems to "want" things that are necessary for both client and server support.

So the other workaround is to simply do this on your machine:

    # systemctl enable nfs-secure.service
    # systemctl enable nfs.target
    # systemctl start nfs.target

...that should get rpc.gssd running and keep it starting up after reboots.

New installations get nfs.target enabled by default, but package upgrades don't. Maybe that ought to be fixed somehow, but it's difficult to tell the difference between "nfs.target was explicitly disabled on this system" and "nfs.target didn't exist at some point in the past and so was never enabled before"

We could, for an interim period simply have the %post section unconditionally reenable nfs.target. We might get some complaints but probably less than the people complaining about this damnable delay when mounting.

There is one change that I think we do need to make regardless. I think we also need to ensure that rpc.gssd is now started by default on new installations. The kernel now upcalls for GSSAPI creds on almost every mount, and if this daemon isn't up you'll get the long delay you're seeing on the first mount attempt.

This patch should fix up the immediate issue, but I haven't had a chance to test it yet.
Comment 15 JM 2013-08-17 13:24:42 UTC 
After the explanation from comment #14 I checked 

/etc/systemd/system/nfs.target.wants

and saw that nfs-idmap.service was part of nfs.target.wants on my system (after the upgrade from F18 to F19) which is not necessary for a client system. So I disabled everything inside /etc/systemd/system/nfs.target.wants enabled only nfs-secure.service and nfs.target and now everything works :-). rpc.gssd is running after the reboot (and only this daemon, not the NFS-Server stuff).

This solved the problem for me, I guess the idea to change the description (nfs.target) to

Network File System Client and Server Support

is good because nfs.target is for the client and server and to enable nfs.target and nfs-secure.service by default is also a good idea, this way the long delay is gone.

Thank you for the help.
Comment 16 antokarag 2013-08-26 17:39:14 UTC 
(In reply to Dean Hunter from comment #4)
> Oops, the workaround should be:
> 
>   cat /usr/lib/systemd/system/nfs-secure.service \
>     | sed -e s/WantedBy=nfs.target/WantedBy=multi-user.target/ \
>     > /etc/systemd/system/nfs-secure.service

Even with this workaround and the fact that nfs-secure.service starts I found a problem. If I try to mount a krb5 nfs4 share, the gssrpc crashes.

If I then do a #systemctl start nfs-secure.service
and retry to mount the share, it works.

The mount fails even if I run it manually (also fails from fstab).
If I perform a #systemctl restart nfs-secure.service
prior the initial mount, it then succeeds. So the problem must be on the init of the service... It does not init properly by itself or the init levels are invalid.