Bug 1189767

Please provide a full debug log.

That means set logging to debug in the autofs configuration
and ensure that syslog is recording debug level messages and
above for facility daemon.

This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Experienced the problem with RHEL 7.0. Upon on reboot from a fresh installation, no nfs share can be auto mounted.

kernel-3.10.0-123.20.1.el7.x86_64
autofs-5.0.7-40.el7.x86_64

# systemctl status autofs -l
autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled)
   Active: active (running) since Mon 2015-03-09 14:09:49 PDT; 24min ago
  Process: 1756 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS)
 Main PID: 1779 (automount)
   CGroup: /system.slice/autofs.service
           └─1779 /usr/sbin/automount -DOSNAME=centos -DOSREL=7 --pid-file /run/autofs.pid

Mar 09 14:09:49 node.example.com systemd[1]: Starting Automounts filesystems on demand...
Mar 09 14:09:49 node.example.com automount[1779]: setautomntent: lookup(sss): setautomntent: No such file or directory
Mar 09 14:09:49 node.example.com systemd[1]: Started Automounts filesystems on demand.

Autofs unit file:

# cat /etc/systemd/system/multi-user.target.wants/autofs.service
[Unit]
Description=Automounts filesystems on demand
After=network.target ypbind.service sssd.service

[Service]
Type=forking
PIDFile=/run/autofs.pid
EnvironmentFile=-/etc/sysconfig/autofs
ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid
ExecReload=/usr/bin/kill -HUP $MAINPID
TimeoutSec=180

[Install]
WantedBy=multi-user.target

The cause for my problem is the network link is not up/ready before SSSD and autofs start.

Mar  9 14:39:02 node sssd: Starting up
....
Mar  9 14:39:05 node automount[1781]: Starting automounter version 5.0.7-40.el7, master map auto.master
....
Mar  9 14:39:06 node kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
...

Autofs pulls automount maps via SSSD, which relies on network connection to the LDAP servers.

So autofs startup should check network link is up and ready. 

The unit file requirement for network is not satisfied and but still autofs got started. This is a serious problem.

Created attachment 999687 [details]
Autofs boot log

(In reply to Simon Gao from comment #5)
> Created attachment 999687 [details]
> Autofs boot log

OK, that might help.

I think there's a bit of a catch 22 here.

On one hand we need a map re-read to continue and use existing
map information so the master map read returns success if at
least one read map succeeds. But at startup it's not so clear,
it might be the case we want to wait for all the master map
entries to become available but equally we might not when it's
expected one or other map will not be readable. There isn't a
way for autofs to know which is which.

I'll have a look around.

Ian

(In reply to Simon Gao from comment #4)
> 
> Autofs pulls automount maps via SSSD, which relies on network connection to
> the LDAP servers.
> 
> So autofs startup should check network link is up and ready. 
> 
> The unit file requirement for network is not satisfied and but still autofs
> got started. This is a serious problem.

No, autofs doesn't know details about the network configuration
and nor should it.

Other units are responsible for that, such as Network Manager,
which can be configured to wait for the network to become
available. The Network Manager folks won't make that the default,
understandably, but perhaps it should be an option in the GUI.

But the bigger question is autofs knowing when a master map read
failure should cause the overall master map read to fail.

I just upgraded a system from Fedora 21 to Fedora 22 and encountered this issue (it wasn't a problem on Fedora 21).

From what I can see in the logs, sssd started before NetworkManager brought the network interface up, so the LDAP server I have configured was "offline" to sssd because the network interface was down and because named had not started yet, so it could not resolve the name.  named and autofs were started after the network interface came up, but since sssd thought that the LDAP server was offline autofs could not get the automount information, and user directories were not available.

I can get around the name resolution issue by putting an entry in /etc/hosts, but it seems like this is an issue with what order things are started up.

If I log in to the system via ssh and restart autofs the problem goes away.  I'm guessing that makes sssd recheck the LDAP server, that works, and then autofs can get the information it needs to automount directories.

I get this pretty often on my network of a hundred+ F23 systems, all fully updated.  Sometimes users reboot their desktops for whatever reason, and autofs just doesn't start.  For whatever reason this often seems to persist after a reboot.  I don't recall seeing the issue all that much back when everything was running F21, but maybe my memory is just poor.

I would hope that SSSD would be able to cache the master automount map so that the ldap servers or the network in general don't need to be up in order for autofs to start, but maybe I'm thinking the issue is simpler than it really is.

I have the same problem on a F22 box.

After boot, none of the automounts are accessible and checking the status of autofs reports "setautomntent: No such file or directory".

Restarting autofs resolves automounts not being accessible and I suspect the network is not up by the time autofs start.  In my case, this machine is using a WiFi connection with WPA-Enterprise (EAP TLS).


# systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-01 01:53:03 CDT; 35min ago
 Main PID: 2202 (automount)
   CGroup: /system.slice/autofs.service
           └─2202 /usr/sbin/automount -t 60 --pid-file /run/autofs.pid

Apr 01 01:53:02 wk07.example.com systemd[1]: Starting Automounts filesystems on demand...
Apr 01 01:53:03 wk07.example.com automount[2202]: setautomntent: lookup(sss): setautomntent: No such file or directory
Apr 01 01:53:03 wk07.example.com systemd[1]: Started Automounts filesystems on demand.

[root@wk07 ~]# systemctl restart autofs

[root@wk07 ~]# systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-01 02:28:29 CDT; 3s ago
  Process: 25148 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS)
 Main PID: 25151 (automount)
   CGroup: /system.slice/autofs.service
           └─25151 /usr/sbin/automount -t 60 --pid-file /run/autofs.pid

Apr 01 02:28:28 wk07.example.com systemd[1]: Starting Automounts filesystems on demand...
Apr 01 02:28:29 wk07.example.com systemd[1]: Started Automounts filesystems on demand.

[root@wk07 ~]# lsb_release -ds
"Fedora release 22 (Twenty Two)"

I have the same problem on F24 box, but did not have it on F22/F23. Should I open a new bug or provide the info here?

I have the same problem with F24 on a virtual machine. Seems to be a timing problem.

Workaround:

cp /usr/lib/systemd/system/sssd.service /etc/systemd/system/sssd.service

And insert ExecStartPre=/bin/sleep 10 in /etc/systemd/system/sssd.service
You may need to play with the value, 10 was okay for me.

The file will look like this:


[Unit]
Description=System Security Services Daemon
# SSSD must be running before we permit user sessions
Before=systemd-user-sessions.service nss-user-lookup.target
Wants=nss-user-lookup.target

[Service]
EnvironmentFile=-/etc/sysconfig/sssd
ExecStartPre=/bin/sleep 10
ExecStart=/usr/sbin/sssd -D -f
# These two should be used with traditional UNIX forking daemons
# consult systemd.service(5) for more details
Type=forking
PIDFile=/var/run/sssd.pid

[Install]
WantedBy=multi-user.target

Ensure the symlinks are correct:

systemctl disable sssd.service
systemctl enable sssd.service

(In reply to Luc de Louw from comment #12)
> I have the same problem with F24 on a virtual machine. Seems to be a timing
> problem.

Thanks, will gladly try this. Still, a proper fix would be nicer.

I have given up on storing the master map in ldap.  I almost never change it, and any change would require a reload of autofs anyway.

I still keep the rest of the maps in ldap, of course, and sssd caches them fine.  Things work even when the machine is booted without the network and it's connected long after autofs has started.

According to the SSSD developers, it should be able to return a cached version of the master map to autofs when the network is not available, but that just doesn't seem to work.

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

This issue is still present in Fedora 24 with:

autofs-5.1.1-22.fc24.x86_64
sssd-1.14.1-2.fc24.x86_64

The workaround in comment 12 still works.

I've been running with a patch from https://fedorahosted.org/sssd/ticket/3080 and it's completely fixed the issue for me.  I imagine that it should make it into the next sssd point release.

(In reply to Jason Bradley Nance from comment #16)
> This issue is still present in Fedora 24 with:
> 
> autofs-5.1.1-22.fc24.x86_64
> sssd-1.14.1-2.fc24.x86_64
> 
> The workaround in comment 12 still works.

Could you test with sssd-1.14.2?
It has fix for https://fedorahosted.org/sssd/ticket/3140
And it should fix the bug.

LS

(In reply to Jason Bradley Nance from comment #16)
> This issue is still present in Fedora 24 with:
> 
> autofs-5.1.1-22.fc24.x86_64
> sssd-1.14.1-2.fc24.x86_64
> 
> The workaround in comment 12 still works.

Could you test with sssd-1.14.2?
It has fix for https://fedorahosted.org/sssd/ticket/3140
And it should fix the bug.

Do not forget to remove workaround before testing

LS

sssd-1.14.2 on F24 appears to be working as expected.


[root@wk07 ~]# lsb_release -ds
"Fedora release 24 (Twenty Four)"

[root@wk07 ~]# rpm -qi sssd
Name        : sssd
Version     : 1.14.2
Release     : 2.fc24
Architecture: x86_64
Install Date: Sun 25 Dec 2016 02:33:12 PM CST

[root@wk07 ~]# systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-01-19 18:08:54 CST; 34min ago
 Main PID: 3601 (automount)
    Tasks: 7 (limit: 512)
   CGroup: /system.slice/autofs.service
           └─3601 /usr/sbin/automount --pid-file /run/autofs.pid

Jan 19 18:08:54 wk07.example.com systemd[1]: Starting Automounts filesystems on demand...
Jan 19 18:08:54 wk07.example.com systemd[1]: Started Automounts filesystems on demand.

[root@wk07 ~]# automount -m

autofs dump map information
===========================

global options: none configured

Mount point: /misc

source(s):

  instance type(s): file 
  map: /etc/auto.misc

  cd | -fstype=iso9660,ro,nosuid,nodev	:/dev/cdrom


Mount point: /-

source(s):

  instance type(s): file 
  map: /etc/auto.direct

  no keys found in map


Mount point: /net

source(s):

  type: hosts


Mount point: /mnt/iso

source(s):

  instance type(s): sss 
  map: auto.isos

  CentOS-6.6-i386-minimal | -fstype=iso9660,loop,ro :/srv/data/pub/linux/CentOS/6.6/isos/i386/CentOS-6.6-i386-minimal.iso
  Fedora-Live-Desktop-i686-20-1 | -fstype=iso9660,loop,ro :/srv/data/pub/linux/Fedora/releases/20/isos/i686/Fedora-Live-Desktop-i686-20-1.iso
...

(In reply to Steven W. Elling from comment #20)
> sssd-1.14.2 on F24 appears to be working as expected.
> 
Thank you for confirmation that it is sssd bug https://fedorahosted.org/sssd/ticket/3140

BTW Are you sure you are not using a workaround from comment 12?

Works for me with in Fedora 25

autofs-5.1.2-1.fc25.x86_64
libsss_autofs-1.14.2-2.fc25.x86_64

Thanks

Thank you for confirmation.

I never implemented the workaround from comment 12.  I just dealt with restarting autofs to resolve the issue on as as needed basis.

I'm seeing these on Fedora 25 workstation using NIS:
Mar  1 15:48:35 ourserver automount[3442]: add_host_addrs: hostname lookup failed: Name or service not known
Mar  1 15:48:35 ourserver automount[3442]: setautomntent: lookup(sss): setautomntent: No such file or directory
Mar  1 15:48:35 ourserver automount[3442]: key "config" not found in map source(s).


But sssd is diasbled as we use nscd: 
Mar  1 16:29:18 ourserver sssd: NSCD socket was detected and seems to be configured to cache some of the databases controlled by SSSD [passwd,group,netgroup,services]. It is recommended not to run NSCD in parallel with SSSD, unless NSCD is configured not to cache these.

(In reply to RobbieTheK from comment #25)
> I'm seeing these on Fedora 25 workstation using NIS:
> Mar  1 15:48:35 ourserver automount[3442]: add_host_addrs: hostname lookup
> failed: Name or service not known
> Mar  1 15:48:35 ourserver automount[3442]: setautomntent: lookup(sss):
> setautomntent: No such file or directory
> Mar  1 15:48:35 ourserver automount[3442]: key "config" not found in map
> source(s).
> 
> 
> But sssd is diasbled as we use nscd: 
> Mar  1 16:29:18 ourserver sssd: NSCD socket was detected and seems to be
> configured to cache some of the databases controlled by SSSD
> [passwd,group,netgroup,services]. It is recommended not to run NSCD in
> parallel with SSSD, unless NSCD is configured not to cache these.

This BZ was about something else.
But I assume you have "automount:  files sss" in nsswitch.conf.
It was *probably* added by authconfig because libsss_autofs was installed on system.

Ah indeed there was a sss entry in /etc/nsswitch.conf. Would this also cause the " automount[3442]: add_host_addrs: hostname lookup failed: Name or service not known" error?

(In reply to RobbieTheK from comment #27)
> Ah indeed there was a sss entry in /etc/nsswitch.conf. Would this also cause
> the " automount[3442]: add_host_addrs: hostname lookup failed: Name or
> service not known" error?

No, it shouldn't cause that to happen.

Change order into autofs.service

For Wants settings, move after "Description...." and append ypbind.service
And add line, Requires=network.target rpc-statd.service rpcbind.service
after "Wants" settings line

Before

[Unit]
Description=Automounts filesystems on demand
After=network.target ypbind.service sssd.service
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/autofs.pid
EnvironmentFile=-/etc/sysconfig/autofs
ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid
ExecReload=/usr/bin/kill -HUP $MAINPID
TimeoutSec=180

[Install]
WantedBy=multi-user.target

After

[Unit]
Description=Automounts filesystems on demand
Wants=network-online.target ypbind.service
Requires=network.target rpc-statd.service rpcbind.service
After=network.target ypbind.service sssd.service network-online.target remote-fs.target

[Service]
Type=forking
PIDFile=/run/autofs.pid
EnvironmentFile=-/etc/sysconfig/autofs
ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid
ExecReload=/usr/bin/kill -HUP $MAINPID
TimeoutSec=180

[Install]
WantedBy=multi-user.target

the autofs mount is ok after changing the settings

I'm seeing this again on a just updated fc29

(In reply to Julen Landa Alustiza from comment #30)
> I'm seeing this again on a just updated fc29

What exactly are you seeing, there were a couple of different problems described above.

Sorry, nvm. The problem was a faulty disk on nfs server side :(