Bug 781564

Summary: SSSD Fails to retrieve groups from LDAP
Product: Red Hat Enterprise Linux 6 Reporter: Rene Snajder <rene.snajder>
Component: sssdAssignee: Stephen Gallagher <sgallagh>
Status: CLOSED NOTABUG QA Contact: IDM QE LIST <seceng-idm-qe-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2CC: grajaiya, jgalipea, jhrozek, prc
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-16 13:52:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Rene Snajder 2012-01-13 18:24:31 UTC
Description of problem:

SSSD version 1.5.1.66 introduced a problem for me where it suddenly couldn't find groups from my LDAP server anymore. The config should be OK, as it worked with previous versions. "getent passwd" shows me all local users and users from the LDAP, but "getent group" only listed local users.

After a long long afternoon I finally managed to track down the problem to the LDAP filter applied. Here is the key log output from my openldap log when running "getent group mygroup" (DNs censored):

Jan 13 18:25:55 localhost slapd[30967]: conn=1107 op=12 SRCH base="ou=groups,ou=what,dc=ever,dc=at" scope=2 deref=0 filter="(&(cn=mygroup)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))"
Jan 13 18:25:55 localhost slapd[30967]: conn=1107 op=12 SRCH attr=objectClass cn userPassword gidNumber memberuid modifyTimestamp modifyTimestamp
Jan 13 18:25:55 localhost slapd[30967]: conn=1107 op=12 SEARCH RESULT tag=101 err=0 nentries=0 text=

As you can see it says "nentries=0", which means there were no results.

Manually calling this filter from the command line gives me the same result. BUT, if I remove the last part "!(gidNumber=0)" (and respective brackets around that statement) I will get this output in the ldap log:

Jan 13 18:24:14 localhost slapd[30967]: conn=1117 op=1 SRCH base="ou=groups,ou=what,dc=ever,dc=at" scope=2 deref=0 filter="(&(cn=mygroup)(objectClass=posixGroup)(cn=*)(gidNumber=*))"
Jan 13 18:24:14 localhost slapd[30967]: conn=1117 op=1 SRCH attr=objectClass cn userPassword gidNumber memberUid modifyTimestamp modifyTimestamp
Jan 13 18:24:14 localhost slapd[30967]: conn=1117 op=1 SEARCH RESULT tag=101 err=0 nentries=1 text=

Now we have "nenetries=1" which means there is a result. Therefore I think there is something wrong with the ldap filter, specifically the "!(gidNumber=0)" part.
But I could be wrong about that.


Version-Release number of selected component (if applicable):
sssd 1.5.1.66


How reproducible:

Steps to Reproduce:
1. Setup SSSD to connect to an LDAP server with default scheme, for user and group providing.
2. call "getent group"
  
Actual results:
groups from ldap are not displayed

Expected results:
groups from ldap should be in there

Additional info:
Downgrading TWICE like this:
yum downgrade sssd sssd-client libipa_hbac libipa_hbac-python

I don't recall the in-between version, but i needed to go back to 1.5.1-34.el6_1.3.

Comment 1 Stephen Gallagher 2012-01-13 18:37:53 UTC
Obvious question: is the gidNumber of that group actually zero? If it is, we can't support that. UID and GID 0 on Linux have special meanings (they have root-level privileges). We intentionally disallow managing that from LDAP. UID and GID 0 are left to the /etc/passwd, /etc/shadow and /etc/group files to manage.

Also, this was unclear:
"getent passwd" shows me all local users and users from
the LDAP, but "getent group" only listed local users.


First of all, did you mean ""getent group" only listed local *groups*?

Also, are you running with enumerate=true? The filters you listed are more in tune with 'getent group mygroup' than 'getent group'.

Comment 3 Rene Snajder 2012-01-13 21:30:32 UTC
No the gidNumber isnt 0. There are about 30 different groups in there, all with gidNumber higher than 20000. And yes, min and max id are set.

Yes I meant getent group listed only local groups.

Enumerate is set. The filters are for "getent group mygroup", which returns nothing. Calling "getent group" returns only groups from the passwd file.

I can post additional logs and config on Monday if you need further information.

Comment 4 Stephen Gallagher 2012-01-13 21:58:50 UTC
Yes, debug logs at level 7 or higher would be great, as well as your sssd.conf. Thank you.

Comment 5 Rene Snajder 2012-01-16 08:23:12 UTC
Hi,

Here is the sssd.conf. We are using ldap for id providing and a custom pam proxy module for authentication.

#-----------------------
[sssd]
config_file_version = 2
reconnection_retries = 3
services = nss, pam
domains = default

[nss]
filter_groups = root
filter_users = root
reconnection_retries = 3

[pam]
reconnection_retries = 3

[domain/default]
auth_provider = proxy
cache_credentials = True
ldap_id_use_start_tls = True
ldap_tls_reqcert = never 
ldap_tls_cacert = /path/to/my/certificate
debug_level = 10
debug-to-files = True
ldap_search_base = dc=mydomain,dc=at
ldap_user_search_base = ou=users,ou=myserver,dc=mydomain,dc=at
ldap_group_search_base = ou=groups,ou=myserver,dc=mydomain,dc=at
id_provider = ldap
min_id = 0
max_id = 99999
ldap_uri = ldaps://127.0.0.1/
ldap_default_bind_dn = UID=myuser,OU=dsa,OU=myserver,DC=mydomain,DC=at
ldap_default_authtok_type = password
ldap_default_authtok = mypassword
chpass_provider = ldap
enumerate = true
proxy_pam_target = mypam
#-----------------------------

And here is a level 10 log as it was written after calling "getent group mygroup":

#-----------------------------
[sssd[be[default]]] [sbus_message_handler] (9): Received SBUS method [getAccountInfo]
[sssd[be[default]]] [be_get_account_info] (4): Got request for [4098][1][name=mygroup]
[sssd[be[default]]] [sdap_id_op_connect_step] (9): reusing cached connection
[sssd[be[default]]] [sdap_get_generic_step] (6): calling ldap_search_ext with [(&(cn=mygroup)(objectclass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][ou=myserver,dc=mydomain,dc=at].
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [objectClass]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [cn]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [userPassword]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [gidNumber]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [memberUid]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [nsUniqueId]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [modifyTimestamp]
[sssd[be[default]]] [sdap_get_generic_step] (7): Requesting attrs: [modifyTimestamp]
[sssd[be[default]]] [sdap_get_generic_step] (8): ldap_search_ext called, msgid = 40
[sssd[be[default]]] [sdap_process_result] (8): Trace: sh[0x87c7b0], connected[1], ops[0x87ca50], ldap[0x878420]
[sssd[be[default]]] [sdap_get_generic_done] (6): Search result: Success(0), (null)
[sssd[be[default]]] [sdap_get_generic_done] (7): Total count [0]
[sssd[be[default]]] [sdap_get_groups_process] (6): Search for groups, returned 0 results.
[sssd[be[default]]] [sdap_id_op_done] (9): releasing operation connection
[sssd[be[default]]] [ldb] (9): tevent: Added timed event "ltdb_callback": 0x878a20
[sssd[be[default]]] [ldb] (9): tevent: Added timed event "ltdb_timeout": 0x878b40
[sssd[be[default]]] [ldb] (9): tevent: Destroying timer event 0x878b40 "ltdb_timeout"
[sssd[be[default]]] [ldb] (9): tevent: Ending timer event 0x878a20 "ltdb_callback"
[sssd[be[default]]] [sysdb_search_group_by_name] (6): Error: 2 (No such file or directory)
[sssd[be[default]]] [sysdb_delete_group] (6): Error: 2 (No such file or directory)
[sssd[be[default]]] [acctinfo_callback] (4): Request processed. Returned 0,0,Success
[sssd[be[default]]] [sdap_process_result] (8): Trace: sh[0x87c7b0], connected[1], ops[(nil)], ldap[0x878420]
[sssd[be[default]]] [sdap_process_result] (8): Trace: ldap_result found nothing!
#-----------------------------

Comment 6 Rene Snajder 2012-01-16 08:27:38 UTC
By the way, this is what the group looks like in LDAP (censored of course):

# mygroup, groups, myserver, mydomain.at
dn: cn=mygroup,ou=groups,ou=myserver,dc=mydomain,dc=at
objectClass: posixGroup
cn: mygrpoup
sambaSID: S-1-5-21-1916803321-1212439250-3036017952-47165
gidNumber: 23082
displayName: My group
sambaGroupType: 2

Comment 7 Jakub Hrozek 2012-01-16 11:51:36 UTC
Rene,

can you try running the following on your client system:

ldapsearch -H ldaps://127.0.0.1/ -b ou=myserver,dc=mydomain,dc=at -D "UID=myuser,OU=dsa,OU=myserver,DC=mydomain,DC=at" -w mypassword "(&(cn=mygroup)(objectclass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))"

The search is based on your sanitized debug output and the config file and it's also what SSSD is doing internally.

I also assume that user information are still returned correctly?

Unrelated question - why are you using the proxy auth target? Are you using some custom authentication method that would prevent you from using auth=ldap?

Comment 8 Rene Snajder 2012-01-16 12:24:54 UTC
Hi,

This is exactly what I tried (as I briefly mentioned in the first post), but I failed to mention the resulting error. My apologies.

If I run this I get:

#-----------------------
# extended LDIF
#
# LDAPv3
# base <ou=myserver,dc=mydomain,dc=at> with scope subtree
# filter: (&(cn=mygroup)(objectclass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))
# requesting: ALL
#

# search result
search: 2
result: 80 Other (e.g., implementation specific) error

# numResponses: 1
#-----------------------

And in the slapd.log:

#-----------------------
slapd[2612]: conn=1011 op=124 SRCH base="ou=users,ou=myserver,dc=mydomain,dc=at" scope=2 deref=0 filter="(&(objectClass=posixAccount)(uid=*)(uidNumber=*)(gidNumber=*))"
slapd[2612]: conn=1011 op=124 SRCH attr=objectClass uid userPassword uidNumber gidNumber gecos homeDirectory loginShell krbPrincipalName cn modifyTimestamp modifyTimestamp shadowLastChange shadowMin shadowMax shadowWarning shadowInactive shadowExpire shadowFlag krbLastPwdChange krbPasswordExpiration pwdAttribute authorizedService accountExpires userAccountControl nsAccountLock
slapd[2612]: conn=1011 op=124 SEARCH RESULT tag=101 err=0 nentries=50 text=
slapd[2612]: conn=1011 op=125 SRCH base="ou=groups,ou=myserver,dc=mydomain,dc=at" scope=2 deref=0 filter="(&(objectClass=posixGroup)(cn=*)(gidNumber=*))"
slapd[2612]: conn=1011 op=125 SRCH attr=objectClass cn userPassword gidNumber memberuid modifyTimestamp modifyTimestamp
slapd[2612]: conn=1011 op=125 SEARCH RESULT tag=101 err=0 nentries=72 text=
#-----------------------

User information is returned properly, yes. Just groups don't seem to work anymore.

About using a proxy auth target: Long story short, depending on which user it is, we have to decide to authenticate to one of two LDAP servers. We have a custom PAM module for that. Its a little bit ugly, but it was necessary.

Comment 9 Stephen Gallagher 2012-01-16 12:36:49 UTC
(In reply to comment #8)

> # search result
> search: 2
> result: 80 Other (e.g., implementation specific) error
> 
> # numResponses: 1
> #-----------------------

This is really telling. Result code "80" implies that something on the server failed. (This would obviously explain the failure in SSSD). If I had to guess, I'd presume that maybe the ACIs on the LDAP server allow checking for gidNumber=* but forbid searches on specific GID values?

It would be interesting to try:
dapsearch -H ldaps://127.0.0.1/ -b ou=myserver,dc=mydomain,dc=at -D
"UID=myuser,OU=dsa,OU=myserver,DC=mydomain,DC=at" -w mypassword
"(&(gidNumber=23082)(objectclass=posixGroup)(cn=*))"

(This will attempt to look up the group by a specific GID).

> 
> And in the slapd.log:
> 
> #-----------------------
> slapd[2612]: conn=1011 op=124 SRCH
> base="ou=users,ou=myserver,dc=mydomain,dc=at" scope=2 deref=0
> filter="(&(objectClass=posixAccount)(uid=*)(uidNumber=*)(gidNumber=*))"
> slapd[2612]: conn=1011 op=124 SRCH attr=objectClass uid userPassword uidNumber
> gidNumber gecos homeDirectory loginShell krbPrincipalName cn modifyTimestamp
> modifyTimestamp shadowLastChange shadowMin shadowMax shadowWarning
> shadowInactive shadowExpire shadowFlag krbLastPwdChange krbPasswordExpiration
> pwdAttribute authorizedService accountExpires userAccountControl nsAccountLock
> slapd[2612]: conn=1011 op=124 SEARCH RESULT tag=101 err=0 nentries=50 text=
> slapd[2612]: conn=1011 op=125 SRCH
> base="ou=groups,ou=myserver,dc=mydomain,dc=at" scope=2 deref=0
> filter="(&(objectClass=posixGroup)(cn=*)(gidNumber=*))"
> slapd[2612]: conn=1011 op=125 SRCH attr=objectClass cn userPassword gidNumber
> memberuid modifyTimestamp modifyTimestamp
> slapd[2612]: conn=1011 op=125 SEARCH RESULT tag=101 err=0 nentries=72 text=
> #-----------------------
> 

This is not the matching slapd log to the failure. The filter is missing the !gidNumber=0 portion. Please include the correct log so we can see if an error is reported.

> About using a proxy auth target: Long story short, depending on which user it
> is, we have to decide to authenticate to one of two LDAP servers. We have a
> custom PAM module for that. Its a little bit ugly, but it was necessary.

Depending on how you determine which user is which, you might be able (in SSSD 1.7.0 and later, coming in 6.3) to set up two separate domains with user filters. If a user is found in DomainA, it will auth against the LDAP server in DomainA. If it's found in DomainB, it will auth against the LDAP server in DomainB. (This assumes of course that which auth server to use is easily identifiable by LDAP search filter).

Comment 10 Rene Snajder 2012-01-16 13:02:44 UTC
Here is the (correct) part of the log file:

#------------------------
slapd[2612]: conn=1158 fd=10 ACCEPT from IP=127.0.0.1:49242 (IP=0.0.0.0:636)
slapd[2612]: conn=1158 fd=10 TLS established tls_ssf=256 ssf=256
slapd[2612]: conn=1158 op=0 BIND dn="uid=myuser,ou=dsa,ou=myserver,dc=mydomain,dc=at" method=128
slapd[2612]: conn=1158 op=0 BIND dn="uid=myuser,ou=dsa,ou=myserver,dc=mydomain,dc=at" mech=SIMPLE ssf=0
slapd[2612]: conn=1158 op=0 RESULT tag=97 err=0 text=
slapd[2612]: conn=1158 op=1 SRCH base="ou=myserver,dc=mydomain,dc=at" scope=2 deref=0 filter="(&(cn=myproject)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))"
slapd[2612]: conn=1158 op=1 SEARCH RESULT tag=101 err=80 nentries=0 text=
slapd[2612]: conn=1158 op=2 UNBIND
slapd[2612]: conn=1158 fd=10 closed
#------------------------

Interesting. I ran the command you told me and I get the same error.  (log looks very similar, no need to attach it).

Whats strange is, if you look at the first post, the ldap log says "err=0" (which was from sssd's call), while calling the filter manually (with ldapsearch) always shows "err=80".

> I'd presume that maybe the ACIs on the LDAP server allow checking for
> gidNumber=* but forbid searches on specific GID values?

While this would certainly fit, I tried to remove all ACLs from the LDAP config and the result is still the same.

> Depending on how you determine which user is which, you might be able (in SSSD
> 1.7.0 and later, coming in 6.3) 

Cool, looking forward to this update!

Comment 11 Stephen Gallagher 2012-01-16 13:15:31 UTC
Would you mind running the ldapsearch I showed you above (the one that resulting in the 80 error code) but add '-d 65535' to the options?

I want to know if the debugging output will reveal more about the specific error we're hitting. (Though I'm not hopeful. I would think that the server log file's 'text=' line would have said).

Whatever is happening here, it's definitely a server-side issue. There's nothing wrong with the ldap search filter here. The inclusion of the exemption for gidNumber=0 is intentional and we don't want to remove it.

Comment 12 Rene Snajder 2012-01-16 13:47:05 UTC
> Would you mind running the ldapsearch I showed you above (the one that
> resulting in the 80 error code) but add '-d 65535' to the options?

I tried it, but won't post the output here, as it will be difficult to censor my ldap setup from it. But it doesn't seem to contain any helpful information.

> Whatever is happening here, it's definitely a server-side issue.

Agreed. I will look into the LDAP setup and try to figure out what's going on there. I'll reopen the bug report if I have further proof that the problem is within SSSD. Thanks for your help.

Comment 13 Stephen Gallagher 2012-01-16 13:52:48 UTC
Thank you and good luck tracking it down.

Closing as NOTABUG.