the sssd_ifp process was supposed to have reconnected on receiving the signal.

is sssd_ifp running at all? Can you enable debugging in the ifp section and attach the debug logs?

(In reply to jared jennings from comment #0)
> 
> Error dbus calling GetUserGroups(myusername):
> org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes
> include: the remote application did not send a reply, the message bus
> security policy blocked the reply, the reply timeout expired, or the network
> connection was broken.

Are there any AVC denials logged in /var/log/audit/audit.log, by any chance?

Could you please open a support case with Red Hat support for proper tracking of the issue and resolution? You can point them to this bugzilla since it captures the technical situation very well but there is a chance the fix might go to some other component than sssd and having a support case will help us to track it.

(In reply to Jan Pazdziora from comment #6)
> Could you please open a support case with Red Hat support 

OK, I'm now the proud owner of case number 01556573.

(In reply to Jan Pazdziora from comment #5)
> Are there any AVC denials logged in /var/log/audit/audit.log, by any chance?

I looked there first, and didn't find any. But now there are some. Here are the ones on the same day as the most recent mod_lookup_identity failure:

2015-12-11 15:21:36 -0500 type=USER_AVC msg=audit(1449865296.587:764): pid=20123 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=3269 tpid=20167 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
2015-12-11 15:32:23 -0500 type=USER_AVC msg=audit(1449865943.132:447): pid=711 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=2719 tpid=3591 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
2015-12-11 15:36:31 -0500 type=USER_AVC msg=audit(1449866191.672:473): pid=711 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=2722 tpid=3728 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
2015-12-11 15:37:21 -0500 type=USER_AVC msg=audit(1449866241.312:496): pid=3773 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=2721 tpid=3780 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'

(I tacked on the human readable times using ruby -ne 'if /\(([0-9.]+):[0-9]*\)/; puts "#{Time.at($1.to_f)} #{$_}"; end')

Each is five seconds before a message in the httpd error log like so - 

/var/log/httpd/foreman-ssl_error_ssl.log-20151214:[Fri Dec 11 15:32:28.139004 2015] [:error] [pid 2719] Error dbus calling GetUserGroups(myusername): org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

(In reply to Jakub Hrozek from comment #1)
> the sssd_ifp process was supposed to have reconnected on receiving the
> signal.
> 
> is sssd_ifp running at all? Can you enable debugging in the ifp section and
> attach the debug logs?

What I saw, as I recall, in the strace, was that the sss_signal signalled the sssd, not sssd_ifp, and there was no sssd_ifp process.

I did enable debugging using sss_debuglevel; I saw the sssd say, "oh someone asked me to reconnect to my sources - ah, look, I'm already connected. OK everything is fine," and nothing about looking up what groups I was in, in response to the request.

Created attachment 1106844 [details]
sssd log with some debugging turned on

OK, here's what I did:

I rotated my logs.

I set the /usr/share/dbus-1/system-services/org.freedesktop.sssd.infopipe.service back to stock (run sss_signal not sssd_ifp), and restarted DBus.

I wrote debug_level=0x3ff in both the [sssd] and [ifp] sections of /etc/sssd/sssd.conf, and restarted sssd.

I logged out and back in to my Satellite server over https, and mod_lookup_identity failed to get my user groups.

/var/log/sssd/sssd_ifp.log stayed at zero size the whole time - its mtime is back in October, which is when I set up this server.

/var/log/sssd/sssd.log contains plenty of stuff, but no 'infopipe' or 'ifp'. Attaching this.

(In reply to jared jennings from comment #8)
> (In reply to Jan Pazdziora from comment #5)
> > Are there any AVC denials logged in /var/log/audit/audit.log, by any chance?
> 
> I looked there first, and didn't find any. But now there are some. Here are
> the ones on the same day as the most recent mod_lookup_identity failure:
> 
> 2015-12-11 15:21:36 -0500 type=USER_AVC msg=audit(1449865296.587:764):
> pid=20123 uid=81 auid=4294967295 ses=4294967295
> subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  {
> send_msg } for msgtype=method_call interface=org.freedesktop.DBus
> member=Hello dest=org.freedesktop.DBus spid=3269 tpid=20167
> scontext=system_u:system_r:httpd_t:s0
> tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus 

Can you do

   ps axuwwfZ | grep sssd

? I'm worried about the tcontext=unconfined_u:unconfined_r:unconfined_t -- sssd should be running as system_u:system_r:sssd_t.

I may well have run the ifp from the command prompt in that instance. I didn't take any notes about what I was trying at what times, and I tried all sorts of stuff.

I've just set the .service file to run sss_signal, rebooted the server entirely, and tailed the audit log while the Satellite login failed. No lines were added. No sssd_ifp process is running, either - neither before nor after the login attempt.

$ ps axuwwfZ | grep sssd
'system_u:system_r:sssd_t:s0     root       736  0.1  0.0 262652  3188 ?        Ss   13:24   0:01 /usr/sbin/sssd -D -f
system_u:system_r:sssd_t:s0     root       737  0.0  0.1 493576 14176 ?        S    13:24   0:00  \_ /usr/libexec/sssd/sssd_be --domain png.loc --uid 0 --gid 0 --debug-to-files
system_u:system_r:sssd_t:s0     root       779  0.0  0.3 268520 29120 ?        S    13:24   0:00  \_ /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
system_u:system_r:sssd_t:s0     root       780  0.0  0.0 246012  4056 ?        S    13:24   0:00  \_ /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 jar2387+ 3987 0.0  0.0 112644 952 pts/0 S+ 13:36   0:00              \_ grep --color=auto sssd

(In reply to jared jennings from comment #12)
> I may well have run the ifp from the command prompt in that instance. I
> didn't take any notes about what I was trying at what times, and I tried all
> sorts of stuff.
> 
> I've just set the .service file to run sss_signal, rebooted the server
> entirely, and tailed the audit log while the Satellite login failed. No
> lines were added. No sssd_ifp process is running, either - neither before
> nor after the login attempt.

If you have configured the external authentication using

  https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/User_Guide/sect-Red_Hat_Satellite-User_Guide-AD_direct.html

that

  katello-installer --foreman-ipa-authentication=true

should have enabled [ifp] in /etc/sssd/sssd.conf for you. No manual setup of .service is needed.

Did you run

  katello-installer --foreman-ipa-authentication=true

?

What is the resulting /etc/sssd/sssd.conf?

Jan, the part I mentioned about the .service file is because I had messed with it before; I was saying that I made sure to put it back to what it says as of https://git.fedorahosted.org/cgit/sssd.git/commit/src?h=sssd-1-13&id=1a59af8245f183f22d87d067a90197d8e2ea958d (cited above).

I did run katello-installer --foreman-ipa-authentication=true. My present sssd.conf is as follows (sanitized):



[sssd]
domains = example.com
config_file_version = 2
services = nss, pam
debug_level=0x3ff

[ifp]
allowed_uids=apache, root
user_attributes=+email, +firstname, +lastname
debug_level=0x3ff

[domain/example.com]
ad_domain = example.com
krb5_realm = EXAMPLE.COM
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = simple
simple_allow_users = myusername, anotherguy

(In reply to jared jennings from comment #14)
> 
> [domain/example.com]
> ad_domain = example.com
> krb5_realm = EXAMPLE.COM
> realmd_tags = manages-system joined-with-samba
> cache_credentials = True
> id_provider = ad
> krb5_store_password_if_offline = True
> default_shell = /bin/bash
> ldap_id_mapping = True
> use_fully_qualified_names = True
> fallback_homedir = /home/%u@%d
> access_provider = simple
> simple_allow_users = myusername, anotherguy

The domain/* section seems to be missing the ldap_user_extra_attrs value.

Added the following to the domain/* section:
ldap_user_extra_attrs = email, firstname, lastname

Restarted sssd, no change in behavior; restarted the whole server, no change in behavior - i.e., there is still no ifp process running, the dbus-send command times out activating the infopipe service, and Foreman fails to ascertain my group membership via mod_lookup_identity. No AVCs in audit log; audit2allow -a -l says nothing. According to the straces, the message causes the dbus daemon to run sss_signal, which reads /var/run/sssd.pid and sends a SIGUSR2 to the pid therein (i.e., the sssd, not the sssd_ifp.)

What I think I'm hearing is that the ifp is supposed to be running the whole time; I'm going to try to find out why it isn't.

I added ifp to the services in the [sssd] section of /etc/sssd/sssd.conf and now sssd_ifp starts when sssd does and everything works right. If there is a bug, which looks unlikely now, my experiences lack the repeatability to help fix it. I'm sorry to have wasted everyone's time.

Sorry for the late response, the whole team was on Christmas vacation.

(In reply to jared jennings from comment #17)
> I added ifp to the services in the [sssd] section of /etc/sssd/sssd.conf and
> now sssd_ifp starts when sssd does and everything works right.

Maybe we need to document things better, was there any documentation you followed that did not explicitly list ifp in the list of services?

> If there is a
> bug, which looks unlikely now, my experiences lack the repeatability to help
> fix it. I'm sorry to have wasted everyone's time.

Not at all :-)

I will keep the bug open for a bit to see if we need to fix the docs. Thank you very much for the additional testing.

Looks like everythign works now, therefore I'm closing the bug. Feel free to ping us about the docs.

Once, I managed to get an HTTP/my.host.name service principal name from my Active Directory instance, but all the other times I've tried, owing I think to Windows being slippery with case, I've ended up with http/my.host.name. That necessitated writing "KrbServiceName http" in /etc/httpd/conf.d/05-foreman-ssl.d/auth_kerb.conf, which gets deleted every time I run katello-installer --foreman-ipa-authentication=true. Perhaps others have better luck and don't need to read about this.

I haven't seen any more trouble with the sssd_ifp as above.

These are all I have to offer in the way of documentation suggestions. Thank you again for your time and courtesy.