Bug 1290910 - sssd-dbus 1.13.0 appears to break mod_lookup_identity
sssd-dbus 1.13.0 appears to break mod_lookup_identity
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sssd (Show other bugs)
7.2
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: SSSD Maintainers
Steeve Goveas
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-11 16:20 EST by jared jennings
Modified: 2016-05-16 10:07 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-11 11:21:14 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sssd log with some debugging turned on (84.68 KB, text/plain)
2015-12-17 15:20 EST, jared jennings
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2097511 None None None 2015-12-18 03:48 EST

  None (edit)
Description jared jennings 2015-12-11 16:20:28 EST
Description of problem:
I've got a RHEL 7.2 server with Satellite 6.1. It's joined to an AD domain using realmd. Satellite is using mod_auth_kerb to authenticate users, as per the Satellite User Guide sec. 20.4 [1]. I've added external groups as per sec. 17.2.1, each of which points to an AD group. I've granted permissions inside Satellite solely to groups, never to users. I configured auto-population of users [3] using mod_lookup_identity, which looks up my group membership and email address using sssd's infopipe dbus service. The end result is that, while a Satellite group must be added for each AD group of interest, no users need to be added to Satellite, and membership of users in groups can be managed solely in AD.

Out of all that, the really important bits to the problem are this: I have a web server with sssd suitably configured and mod_lookup_identity enabled. 

When I upgraded sssd-dbus from 1.12.2-58.el7_1.18.x86_64 to 1.13.0-40.el7.x86_64, this error started showing up in /var/log/httpd/foreman-ssl_error_ssl.log:

Error dbus calling GetUserGroups(myusername@example.com): org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Since I've granted permissions in Satellite only to groups, the immediate symptom was that Satellite forgot that I was allowed to do anything. 

I found a much simpler way [4] to make the GetUserGroups query than trying to log into the Satellite website:

sudo dbus-send --print-reply --system --dest=org.freedesktop.sssd.infopipe /org/freedesktop/sssd/infopipe org.freedesktop.sssd.infopipe.GetUserGroups string:myusername@example.com

When this works right, the results look like:

method return sender=:1.4 -> dest=:1.3 reply_serial=2
   array [
      string "some group i am in"
      string "domain users"
      string "another group"
      string "etc etc"
   ]

When it fails, it looks like this:

Error org.freedesktop.DBus.Error.TimedOut: Activation of org.freedesktop.sssd.infopipe timed out

I ran strace -ff on my dbus daemon, and saw it run sss-signal immediately when the GetUserGroups message was received. sss-signal sent a Unix signal to an sssd process and exited, never having done anything at all with the incoming message.

The service file that compels this behavior was added in July 2014 [5] and first released in sssd 1.30; the sss-signal program appears to be intended to tell sssd to get online with its providers.

When I changed the service file to run sssd_ifp instead of sss-signal, everything started working properly. I won't say it's a fix, because there must have been some case which necessitated the creation and use of the sss-signal program. But it seems to be a suitable temporary workaround at this site.

When sssd was updated a couple of days ago [6], I upgraded sssd-dbus from 1.13.0-40.el7.x86_64 to 1.13.0-40.el7_2.1.x86_64. It broke again in the same way, and I had to do the same workaround.


Links:
[1] Satellite User Guide, sec. 20.4, Using Active Directory Directly: https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/User_Guide/sect-Red_Hat_Satellite-User_Guide-AD_direct.html
[2] Satellite User Guide, sec. 17.2.1, Configuring External User Groups: https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/User_Guide/sect-Red_Hat_Satellite-User_Guide-Users_and_Roles-Creating_User_Groups.html#sect-Red_Hat_Satellite-User_Guide-Creating_User_Groups-Configuring_External_User_Groups
[3] Foreman 1.7 Manual, sec. 5.7.5, Populate users and attributes: http://theforeman.org/manuals/1.7/index.html#5.7.5Populateusersandattributes
[4] mod_lookup_identity: http://adam.younglogic.com/2014/05/mod_lookup_identity/
[5] Commit "Add the DBus service activation": https://git.fedorahosted.org/cgit/sssd.git/commit/src?h=sssd-1-13&id=1a59af8245f183f22d87d067a90197d8e2ea958d
[6] RHBA-2015:2564: https://rhn.redhat.com/errata/RHBA-2015-2564.html?sc_cid=701600000006NHXAA2
Comment 1 Jakub Hrozek 2015-12-13 06:57:10 EST
the sssd_ifp process was supposed to have reconnected on receiving the signal.

is sssd_ifp running at all? Can you enable debugging in the ifp section and attach the debug logs?
Comment 5 Jan Pazdziora 2015-12-17 03:28:03 EST
(In reply to jared jennings from comment #0)
> 
> Error dbus calling GetUserGroups(myusername@example.com):
> org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes
> include: the remote application did not send a reply, the message bus
> security policy blocked the reply, the reply timeout expired, or the network
> connection was broken.

Are there any AVC denials logged in /var/log/audit/audit.log, by any chance?
Comment 6 Jan Pazdziora 2015-12-17 03:51:56 EST
Could you please open a support case with Red Hat support for proper tracking of the issue and resolution? You can point them to this bugzilla since it captures the technical situation very well but there is a chance the fix might go to some other component than sssd and having a support case will help us to track it.
Comment 7 jared jennings 2015-12-17 14:17:22 EST
(In reply to Jan Pazdziora from comment #6)
> Could you please open a support case with Red Hat support 

OK, I'm now the proud owner of case number 01556573.
Comment 8 jared jennings 2015-12-17 14:50:09 EST
(In reply to Jan Pazdziora from comment #5)
> Are there any AVC denials logged in /var/log/audit/audit.log, by any chance?

I looked there first, and didn't find any. But now there are some. Here are the ones on the same day as the most recent mod_lookup_identity failure:

2015-12-11 15:21:36 -0500 type=USER_AVC msg=audit(1449865296.587:764): pid=20123 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=3269 tpid=20167 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
2015-12-11 15:32:23 -0500 type=USER_AVC msg=audit(1449865943.132:447): pid=711 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=2719 tpid=3591 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
2015-12-11 15:36:31 -0500 type=USER_AVC msg=audit(1449866191.672:473): pid=711 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=2722 tpid=3728 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
2015-12-11 15:37:21 -0500 type=USER_AVC msg=audit(1449866241.312:496): pid=3773 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=2721 tpid=3780 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'

(I tacked on the human readable times using ruby -ne 'if /\(([0-9.]+):[0-9]*\)/; puts "#{Time.at($1.to_f)} #{$_}"; end')

Each is five seconds before a message in the httpd error log like so - 

/var/log/httpd/foreman-ssl_error_ssl.log-20151214:[Fri Dec 11 15:32:28.139004 2015] [:error] [pid 2719] Error dbus calling GetUserGroups(myusername@example.com): org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Comment 9 jared jennings 2015-12-17 15:03:35 EST
(In reply to Jakub Hrozek from comment #1)
> the sssd_ifp process was supposed to have reconnected on receiving the
> signal.
> 
> is sssd_ifp running at all? Can you enable debugging in the ifp section and
> attach the debug logs?

What I saw, as I recall, in the strace, was that the sss_signal signalled the sssd, not sssd_ifp, and there was no sssd_ifp process.

I did enable debugging using sss_debuglevel; I saw the sssd say, "oh someone asked me to reconnect to my sources - ah, look, I'm already connected. OK everything is fine," and nothing about looking up what groups I was in, in response to the request.
Comment 10 jared jennings 2015-12-17 15:20 EST
Created attachment 1106844 [details]
sssd log with some debugging turned on

OK, here's what I did:

I rotated my logs.

I set the /usr/share/dbus-1/system-services/org.freedesktop.sssd.infopipe.service back to stock (run sss_signal not sssd_ifp), and restarted DBus.

I wrote debug_level=0x3ff in both the [sssd] and [ifp] sections of /etc/sssd/sssd.conf, and restarted sssd.

I logged out and back in to my Satellite server over https, and mod_lookup_identity failed to get my user groups.

/var/log/sssd/sssd_ifp.log stayed at zero size the whole time - its mtime is back in October, which is when I set up this server.

/var/log/sssd/sssd.log contains plenty of stuff, but no 'infopipe' or 'ifp'. Attaching this.
Comment 11 Jan Pazdziora 2015-12-18 07:38:39 EST
(In reply to jared jennings from comment #8)
> (In reply to Jan Pazdziora from comment #5)
> > Are there any AVC denials logged in /var/log/audit/audit.log, by any chance?
> 
> I looked there first, and didn't find any. But now there are some. Here are
> the ones on the same day as the most recent mod_lookup_identity failure:
> 
> 2015-12-11 15:21:36 -0500 type=USER_AVC msg=audit(1449865296.587:764):
> pid=20123 uid=81 auid=4294967295 ses=4294967295
> subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  {
> send_msg } for msgtype=method_call interface=org.freedesktop.DBus
> member=Hello dest=org.freedesktop.DBus spid=3269 tpid=20167
> scontext=system_u:system_r:httpd_t:s0
> tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dbus 

Can you do

   ps axuwwfZ | grep sssd

? I'm worried about the tcontext=unconfined_u:unconfined_r:unconfined_t -- sssd should be running as system_u:system_r:sssd_t.
Comment 12 jared jennings 2015-12-18 14:26:22 EST
I may well have run the ifp from the command prompt in that instance. I didn't take any notes about what I was trying at what times, and I tried all sorts of stuff.

I've just set the .service file to run sss_signal, rebooted the server entirely, and tailed the audit log while the Satellite login failed. No lines were added. No sssd_ifp process is running, either - neither before nor after the login attempt.

$ ps axuwwfZ | grep sssd
'system_u:system_r:sssd_t:s0     root       736  0.1  0.0 262652  3188 ?        Ss   13:24   0:01 /usr/sbin/sssd -D -f
system_u:system_r:sssd_t:s0     root       737  0.0  0.1 493576 14176 ?        S    13:24   0:00  \_ /usr/libexec/sssd/sssd_be --domain png.loc --uid 0 --gid 0 --debug-to-files
system_u:system_r:sssd_t:s0     root       779  0.0  0.3 268520 29120 ?        S    13:24   0:00  \_ /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
system_u:system_r:sssd_t:s0     root       780  0.0  0.0 246012  4056 ?        S    13:24   0:00  \_ /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 jar2387+ 3987 0.0  0.0 112644 952 pts/0 S+ 13:36   0:00              \_ grep --color=auto sssd
Comment 13 Jan Pazdziora 2015-12-18 14:50:02 EST
(In reply to jared jennings from comment #12)
> I may well have run the ifp from the command prompt in that instance. I
> didn't take any notes about what I was trying at what times, and I tried all
> sorts of stuff.
> 
> I've just set the .service file to run sss_signal, rebooted the server
> entirely, and tailed the audit log while the Satellite login failed. No
> lines were added. No sssd_ifp process is running, either - neither before
> nor after the login attempt.

If you have configured the external authentication using

  https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/User_Guide/sect-Red_Hat_Satellite-User_Guide-AD_direct.html

that

  katello-installer --foreman-ipa-authentication=true

should have enabled [ifp] in /etc/sssd/sssd.conf for you. No manual setup of .service is needed.

Did you run

  katello-installer --foreman-ipa-authentication=true

?

What is the resulting /etc/sssd/sssd.conf?
Comment 14 jared jennings 2015-12-21 11:47:20 EST
Jan, the part I mentioned about the .service file is because I had messed with it before; I was saying that I made sure to put it back to what it says as of https://git.fedorahosted.org/cgit/sssd.git/commit/src?h=sssd-1-13&id=1a59af8245f183f22d87d067a90197d8e2ea958d (cited above).

I did run katello-installer --foreman-ipa-authentication=true. My present sssd.conf is as follows (sanitized):



[sssd]
domains = example.com
config_file_version = 2
services = nss, pam
debug_level=0x3ff

[ifp]
allowed_uids=apache, root
user_attributes=+email, +firstname, +lastname
debug_level=0x3ff

[domain/example.com]
ad_domain = example.com
krb5_realm = EXAMPLE.COM
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = simple
simple_allow_users = myusername, anotherguy
Comment 15 Jan Pazdziora 2015-12-21 13:23:31 EST
(In reply to jared jennings from comment #14)
> 
> [domain/example.com]
> ad_domain = example.com
> krb5_realm = EXAMPLE.COM
> realmd_tags = manages-system joined-with-samba
> cache_credentials = True
> id_provider = ad
> krb5_store_password_if_offline = True
> default_shell = /bin/bash
> ldap_id_mapping = True
> use_fully_qualified_names = True
> fallback_homedir = /home/%u@%d
> access_provider = simple
> simple_allow_users = myusername, anotherguy

The domain/* section seems to be missing the ldap_user_extra_attrs value.
Comment 16 jared jennings 2015-12-21 14:29:36 EST
Added the following to the domain/* section:
ldap_user_extra_attrs = email, firstname, lastname

Restarted sssd, no change in behavior; restarted the whole server, no change in behavior - i.e., there is still no ifp process running, the dbus-send command times out activating the infopipe service, and Foreman fails to ascertain my group membership via mod_lookup_identity. No AVCs in audit log; audit2allow -a -l says nothing. According to the straces, the message causes the dbus daemon to run sss_signal, which reads /var/run/sssd.pid and sends a SIGUSR2 to the pid therein (i.e., the sssd, not the sssd_ifp.)

What I think I'm hearing is that the ifp is supposed to be running the whole time; I'm going to try to find out why it isn't.
Comment 17 jared jennings 2015-12-21 16:30:22 EST
I added ifp to the services in the [sssd] section of /etc/sssd/sssd.conf and now sssd_ifp starts when sssd does and everything works right. If there is a bug, which looks unlikely now, my experiences lack the repeatability to help fix it. I'm sorry to have wasted everyone's time.
Comment 18 Jakub Hrozek 2016-01-04 04:54:48 EST
Sorry for the late response, the whole team was on Christmas vacation.

(In reply to jared jennings from comment #17)
> I added ifp to the services in the [sssd] section of /etc/sssd/sssd.conf and
> now sssd_ifp starts when sssd does and everything works right.

Maybe we need to document things better, was there any documentation you followed that did not explicitly list ifp in the list of services?

> If there is a
> bug, which looks unlikely now, my experiences lack the repeatability to help
> fix it. I'm sorry to have wasted everyone's time.

Not at all :-)

I will keep the bug open for a bit to see if we need to fix the docs. Thank you very much for the additional testing.
Comment 19 Jakub Hrozek 2016-01-11 11:21:14 EST
Looks like everythign works now, therefore I'm closing the bug. Feel free to ping us about the docs.
Comment 20 jared jennings 2016-05-16 10:07:41 EDT
Once, I managed to get an HTTP/my.host.name service principal name from my Active Directory instance, but all the other times I've tried, owing I think to Windows being slippery with case, I've ended up with http/my.host.name. That necessitated writing "KrbServiceName http" in /etc/httpd/conf.d/05-foreman-ssl.d/auth_kerb.conf, which gets deleted every time I run katello-installer --foreman-ipa-authentication=true. Perhaps others have better luck and don't need to read about this.

I haven't seen any more trouble with the sssd_ifp as above.

These are all I have to offer in the way of documentation suggestions. Thank you again for your time and courtesy.

Note You need to log in before you can comment on or make changes to this bug.