Bug 1298115

Description of problem: 

The autofs service fails to load maps on boot if the maps are stored on a NFS mount. 

When a RHEL 7 machine is rebooted the automounted file systems are not available.  Performing a systemctl status -l autofs.service shows that the autofs service could not read the maps.  Once the machine is up performing "systemctl restart autofs.service" makes the automounted file systems available which means the configuration is correct.

Adding "remote-fs.target" to the After option in /usr/lib/systemd/system/autofs.service which appears to have fixed the issue.


Version-Release number of selected component (if applicable):

RHEL 7
autofs-5.0.7-48.el7.x86_64
nfs-utils-1.3.0-0.8.el7.x86_64


How reproducible:

Autofs maps need to be stored on NFS mount.

Steps to Reproduce:

[1] On NFS Client :

[root@dhcp5-3 ~]~]# rpm -qa |grep -i autofs
autofs-5.0.7-48.el7.x86_64

[root@dhcp5-3 ~]~]# rpm -qa |grep -i nfs
libnfsidmap-0.25-11.el7.x86_64
nfs-utils-1.3.0-0.8.el7.x86_64

[2] Made an entry in fstab:

[root@dhcp5-3 ~]~]# mkdir -p /testnfs
[[root@dhcp5-3 ~]~]# cat /etc/fstab
10.65.5.106:/test        /testnfs              nfs      defaults,_netdev    0 0

[3] Mounted the NFS filesystem :

[[root@dhcp5-3 ~]/]# mount -a
[[root@dhcp5-3 ~]/]# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/vda2            9.4G 1023M  8.4G  11% /
devtmpfs             489M     0  489M   0% /dev
tmpfs                497M     0  497M   0% /dev/shm
tmpfs                497M  6.6M  491M   2% /run
tmpfs                497M     0  497M   0% /sys/fs/cgroup
/dev/vda1            473M  117M  357M  25% /boot
10.65.5.106:/test   50G  2.1G   45G   5% /testnfs

[4] Autofs configuration on the same Client:
[root@dhcp5-3 ~]testnfs]# cat /etc/auto.master
#
# Sample auto.master file

# precedence.
#
+auto.master
/-    /testnfs/autofsmount.misc  <<-------- Added this entry . Note autofs map is created on NFS mounted filesystem


[5] Map Configuration :

testnfs]# cat autofsmount.misc
/testautofs    10.65.5.106:/mnt

[6] Testing Autofs mount :

testnfs]# service autofs status
Redirecting to /bin/systemctl status  autofs.service
autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; disabled)
   Active: inactive (dead)

[root@dhcp5-3 ~]/]# systemctl list-unit-files --type service |grep -i autofs
autofs.service                              disabled
[root@dhcp5-3 ~]/]# systemctl enable autofs.service
ln -s '/usr/lib/systemd/system/autofs.service' '/etc/systemd/system/multi-user.target.wants/autofs.service'
[root@dhcp5-3 ~]/]# systemctl start autofs.service
[root@dhcp5-3 ~]/]# systemctl status autofs.service
autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled)
   Active: active (running) since Mon 2016-01-11 12:29:56 IST; 6s ago
  Process: 2813 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS)
 Main PID: 2815 (automount)
   CGroup: /system.slice/autofs.service
           └─2815 /usr/sbin/automount --pid-file /run/autofs.pid

[root@dhcp5-3 ~]/]# cat /proc/mounts |grep -i testautofs
10.65.5.106:/mnt /testautofs nfs4 rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.65.5.3,local_lock=none,addr=10.65.5.106 0 0
[[root@dhcp5-3 ~]/]# cd /testautofs/
[[root@dhcp5-3 ~]testautofs]# ls
aquota.group  test1   test13  test17  test20  test24  test28  test31  test35  test39  test42  test46  test5   test8
aquota.user   test10  test14  test18  test21  test25  test29  test32  test36  test4   test43  test47  test50  test9
example1      test11  test15  test19  test22  test26  test3   test33  test37  test40  test44  test48  test6
lost+found    test12  test16  test2   test23  test27  test30  test34  test38  test41  test45  test49  test7

[7] Rebooted the Client Machine :

[root@dhcp5-3 ~]# uptime
 12:59:37 up 3 min,  1 user,  load average: 0.18, 0.17, 0.07
 
[8] NFS mounted filesystem is seen available :

[root@dhcp5-3 ~]# cat /proc/mounts |grep -i test
 10.65.5.106:/test   50G  2.1G   45G   5% /testnfs

[9] Autofs is not available after reboot:
[root@dhcp5-3 ~]~]# cd /testautofs/
[root@dhcp5-3 ~]testautofs]# ls
testautofs]# systemctl list-unit-files --type service |grep -i autofs
autofs.service                              enabled
testautofs]# systemctl status autofs.service
autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled)
   Active: active (running) since Mon 2016-01-11 12:56:52 IST; 1min 54s ago
  Process: 1078 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS)
 Main PID: 1084 (automount)
   CGroup: /system.slice/autofs.service
           └─1084 /usr/sbin/automount --pid-file /run/autofs.pid

Jan 13 05:38:37 dhcp5-3 automount[1084]: mount_autofs_direct: failed to read direct map
Jan 13 05:38:37 dhcp5-3 automount[1084]: handle_mounts: mount of /- failed!
Jan 13 05:38:37 dhcp5-3 automount[1084]: master_do_mount: failed to startup mount
Jan 13 05:38:37 dhcp5-3 systemd[1]: Started Automounts filesystems on demand.

<Snip from messages file>
Jan 13 05:38:37  dhcp5-3 automount[1086]: file map /testnfs/autofsmount.misc not found
Jan 13 05:38:37  dhcp5-3 automount[1086]: mount_autofs_direct: failed to read direct map
Jan 13 05:38:37  dhcp5-3 automount[1086]: handle_mounts: mount of /- failed!
Jan 13 05:38:37  dhcp5-3 automount[1086]: master_do_mount: failed to startup mount
Jan 13 05:38:37  dhcp5-3 systemd: Started Automounts filesystems on demand.
Jan 13 05:38:37  dhcp5-3 nfsidmap[1137]: nss_getpwnam: name 'root' does not map into domain 'localdomain'
Jan 13 05:38:37  dhcp5-3 systemd: Started Postfix Mail Transport Agent.
Jan 13 05:38:37  dhcp5-3 systemd: Mounted /testnfs.
Jan 13 05:38:37  dhcp5-3 systemd: Starting Remote File Systems.
</Snip>

We can see in the above output of messages file, autofs is started before the NFS filesystem is mounted and hence it is unable to read any maps.

[10] We then changed behaviour of autofs startup to wait till the NFS filesystems are mounted,this can be changed in file : /usr/lib/systemd/system/autofs.service.

testautofs]# systemctl list-dependencies remote-fs.target
remote-fs.target
└─testnfs.mount

testautofs]# cat /usr/lib/systemd/system/autofs.service
[Unit]
Description=Automounts filesystems on demand
After=network.target ypbind.service sssd.service remote-fs.target   <<--------- Added Here  

[Service]
Type=forking
PIDFile=/run/autofs.pid
EnvironmentFile=-/etc/sysconfig/autofs
ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid
ExecReload=/usr/bin/kill -HUP $MAINPID
TimeoutSec=180

[Install]
WantedBy=multi-user.target

[11] Rebooted the system, this time we can see from below messages, that Autofs waits for the NFS filesystem to mounted:

Jan 13 05:47:45 dhcp5-3 systemd: Starting Remote File Systems (Pre).
Jan 13 05:47:45  dhcp5-3 systemd: Reached target Remote File Systems (Pre).
Jan 13 05:47:45  dhcp5-3 systemd: Mounting /testnfs...
Jan 13 05:47:45 dhcp5-3 kernel: FS-Cache: Loaded
Jan 13 05:47:45 dhcp5-3 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 05:47:45 dhcp5-3 kernel: RPC: Registered udp transport module.
Jan 13 05:47:45 dhcp5-3 kernel: RPC: Registered tcp transport module.
Jan 13 05:47:45 dhcp5-3 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 05:47:45 dhcp5-3 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 05:47:45 dhcp5-3 kernel: Key type dns_resolver registered
Jan 13 05:47:45 dhcp5-3 kernel: NFS: Registering the id_resolver key type
Jan 13 05:47:45 dhcp5-3 kernel: Key type id_resolver registered
Jan 13 05:47:45 dhcp5-3 kernel: Key type id_legacy registered
Jan 13 05:47:45 dhcp5-3 nfsidmap[1122]: nss_getpwnam: name 'root' does not map into domain 'localdomain'
Jan 13 05:47:45 dhcp5-3 systemd: Mounted /testnfs.
Jan 13 05:47:45 dhcp5-3 systemd: Starting Remote File Systems.
Jan 13 05:47:45 dhcp5-3 systemd: Reached target Remote File Systems.


Jan 13 05:47:46  dhcp5-3 automount[1154]: mounted indirect on /misc with timeout 300, freq 75 seconds
Jan 13 05:47:46  dhcp5-3 automount[1154]: mounted indirect on /net with timeout 300, freq 75 seconds
Jan 13 05:47:46  dhcp5-3 automount[1154]: mounted direct on /testautofs with timeout 300, freq 75 seconds
Jan 13 05:47:46  dhcp5-3 systemd: Started Automounts filesystems on demand.


[root@dhcp5-3 ~]~]# cd /testautofs/
[root@dhcp5-3 ~]testautofs]# ls
aquota.group  test1   test13  test17  test20  test24  test28  test31  test35  test39  test42  test46  test5   test8
aquota.user   test10  test14  test18  test21  test25  test29  test32  test36  test4   test43  test47  test50  test9
example1      test11  test15  test19  test22  test26  test3   test33  test37  test40  test44  test48  test6
lost+found    test12  test16  test2   test23  test27  test30  test34  test38  test41  test45  test49  test7


[[root@dhcp5-3 ~]/]# uptime
 08:05:06 up 17 min,  1 user,  load average: 0.00, 0.01, 0.02


Expected results:  automounted file systems are not available after client is rebooted.



Additional info: 

From the messages file we could see autofs is started before the NFS filesystem is mounted and hence it is unable to read any maps.

We then changed behaviour of autofs startup to wait till the NFS filesystems are mounted,this can be changed in file : /usr/lib/systemd/system/autofs.service.

testautofs]# cat /usr/lib/systemd/system/autofs.service
[Unit]
Description=Automounts filesystems on demand
After=network.target ypbind.service sssd.service remote-fs.target   <<--------- Added Here

(In reply to Ashima Rawat from comment #1)
> 
> [3] Mounted the NFS filesystem :
> 
> [[root@dhcp5-3 ~]/]# mount -a
> [[root@dhcp5-3 ~]/]# df -h
> Filesystem           Size  Used Avail Use% Mounted on
> /dev/vda2            9.4G 1023M  8.4G  11% /
> devtmpfs             489M     0  489M   0% /dev
> tmpfs                497M     0  497M   0% /dev/shm
> tmpfs                497M  6.6M  491M   2% /run
> tmpfs                497M     0  497M   0% /sys/fs/cgroup
> /dev/vda1            473M  117M  357M  25% /boot
> 10.65.5.106:/test   50G  2.1G   45G   5% /testnfs

Right.

snip ...

> 
> We can see in the above output of messages file, autofs is started before
> the NFS filesystem is mounted and hence it is unable to read any maps.

I get that, ok.

> 
> [10] We then changed behaviour of autofs startup to wait till the NFS
> filesystems are mounted,this can be changed in file :
> /usr/lib/systemd/system/autofs.service.
> 
> testautofs]# systemctl list-dependencies remote-fs.target
> remote-fs.target
> └─testnfs.mount
> 
> testautofs]# cat /usr/lib/systemd/system/autofs.service
> [Unit]
> Description=Automounts filesystems on demand
> After=network.target ypbind.service sssd.service remote-fs.target  
> <<--------- Added Here  

I don't think that adding this dependency to the autofs unit
will will have adverse side effects.

I can add it and hopefully that won't cause other problems
but I can't be certain.

If our regression tests run through without problem that will
give us a reasonable amount of confidence the change won't be
a problem with other setups.

Ian

Hi Ian,

Thanks for such quick response. 

Let me know if you want me to test few more scenarios prior making this change.
Also I shall be communicating the customer about the next action plan on the bugzilla raised to address this issue.

Thanks
Ashima

(In reply to Ashima Rawat from comment #4)
> Hi Ian,
> 
> Thanks for such quick response. 
> 
> Let me know if you want me to test few more scenarios prior making this
> change.
> Also I shall be communicating the customer about the next action plan on the
> bugzilla raised to address this issue.

I think the regression tests we have cover quite a few
scenarios, probably better than we could come up with
anyway.

So I don't think we would gain anything trying to come
up with specific cases. The ones that could cause issue
are probably not easy to produce even if we could think
of some that might be a problem.

We'll need to follow the process which means waiting for
pm and qa acks before this can be go further.

Hi Ian,

Could you please confirm if this issue is going to be fixed in RHEL 7.3 and accordingly I can update the customer ?


Thanks
ashima

(In reply to Ashima Rawat from comment #6)
> Hi Ian,
> 
> Could you please confirm if this issue is going to be fixed in RHEL 7.3 and
> accordingly I can update the customer ?

As in comment #3 I don't think adding the unit dependency
will cause problems and the bug has the needed acks, so
yes it should be included in 7.3.

I do need to try and understand what remote-fs.target does
though, just in case.

Ian

Hi Ian,

Thanks for the clarification. I have updated the customer about the fix to be available in RHEL 7.3 and he is happy to go ahead with case closure for now and states "looking forward to see  fix in RHEL 7.3".

Thanks
Ashima

Created attachment 1154490 [details]
Patch - make service want network-online

This patch adds a systemd dependency to wait for the newtork
to be online.

A similar problem to what has been reported here was reported
against Fedora and this change has been verified to fix that
problem.

But we want the remote file systems target to succeed too (the
next change for this bug) so the network target will need to
have been reached also so it seems sensible to include this
change as well.

Created attachment 1154492 [details]
Patch - add remote-fs.target systemd dependency

Add a dependency for the remote-fs target to the autofs
systemd unit, as described in comment #3.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2508.html