RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1172338 - Why does 'getent group groupname' work quickly but 'id username' (and likewise sss-pam) work very slowly?
Summary: Why does 'getent group groupname' work quickly but 'id username' (and likewis...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: sssd
Version: 6.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: SSSD Maintainers
QA Contact: Kaushik Banerjee
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-09 21:23 UTC by John Dickerson
Modified: 2021-06-10 10:49 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-12-11 09:07:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
A proposed man page patch. (2.22 KB, patch)
2014-12-12 09:41 UTC, Jakub Hrozek
no flags Details | Diff

Description John Dickerson 2014-12-09 21:23:41 UTC
Description of problem:

I have sssd set up to use the 'ad' id_provider.  I'm using auth_provider=simple, with simple_allow_groups=some-big-group
(where some-big-group has around 13,000* users).  

Naturally, I'm also using pam_sss in PAM, and sss in nsswitch.

Everything works fine, it's just that when I do 'getent group some-big-group' from the command line, it returns the entire list of 13,000 members in like .05 seconds. Very fast.  But if I do 'id username', it takes up to 60 seconds to list the groups I belong to (depending on the speed of the AD controller it's connecting to).  

Now, I understand that 'id' needs to grab a whole bunch of groups a user is a member of.  So I'm not really shocked that it takes much longer to get an answer than 'getent group'.   

What seems odd is that 'getent group some-big-group' can list the members so quickly, but pam_sss seems to take a long time to figure out if a user is in the group.  Besides aren't the results from the 'getent group' lookup cached by SSSD?

When I use pam_sss along with the access_provider=simple, and simple_allow_groups=some-big-group, it seems to take pam_sss quite a long time to check if I'm in the group.  It typically takes 20 to 30 seconds after I enter my password in SSH to confirm I'm a member of the group.

Is access_provider=simple really slow?  (I guess I assumed that access_provider=simple uses something like getent to check group membership).

Version-Release number of selected component (if applicable):
sssd-1.11.6-30.el6_6.3.x86_64

How reproducible:
upgrade to latest sssd.  stop sssd.  clear the sssd cache.  restart sssd.

Steps to Reproduce:
1.  getent group some-big-group   (.05 seconds, even with over 13,000 members and a fresh SSSD cache)
2.  configure SSH to use pam_sss
3.  ssh to the system.  It takes up to 45 seconds after I enter my password for it to verify that I'm in the group.

Actual results:
ssh login takes up to 45 seconds for pam_sss to verify that I'm in the group.

Expected results:
I would expect maybe a few seconds for pam_sss to check whether I'm a member of the group, especially since 'getent group some-big-group' works so quickly.

Additional info:

I use the Active Directory for both id_provider and auth_provider.  We have failover configured, and all four of our AD controllers are working fine.

Comment 2 John Dickerson 2014-12-09 22:11:21 UTC
in my original text above I said I use 'auth_provider = simple'.  What I meant to say is 'access_provider = simple'.  

my bad.  

I might also point out that I have tried using 'access_provider = ldap', with an ldap_access_filter set to:
ldap_access_filter = (memberOf:1.2.840.113556.1.4.1941:=CN=engr-lab-acl,OU=UserReqLists,OU=AutoLists,DC=iastate,DC=edu)

This works too, but is no faster than access_provider = simple.  It still takes 30 to 40 seconds to get access.

Finally, I need to mention that after the initial caching is done of the access group in the SSSD cache, subsequent logins are near instantaneous (which is expected) so long as I log in before the cache entry has expired.  Unfortunately, for most users, this means that every time they log in, it takes them 30 to 40 seconds to get logged in.

Comment 3 Jakub Hrozek 2014-12-10 17:16:45 UTC
There might be two things at play. One is that a large group takes a long time to resolve, the other is that during login, we always refresh group memberships from the remote server, bypassing the cache, to make sure access control uses up-to date group information.

The former can be tested quite simply:
for gid in $(id -G $user); do time getent group $gid; done

and check which getgrgid() takes the longest. The ignore_group_members option might come handy.

The latter is something we're going to fix in a later release, to satisfy environments that can tolerate a certain period where stale data might be used and favor speed.

Comment 4 John Dickerson 2014-12-10 18:38:52 UTC
Thanks for the suggestions.

I was never sure what 'ignore_group_members' would do, but I think that is the solution.  If you set "ignore_group_members = true", then group lookups are much much faster in general.  

The only downside is that 'getent group $somegroup' doesn't list the users in the group.  Not a big deal.  In fact, there's an argument to be made that it's better from a security point of view to suppress listing the group members.

So now, instead of taking 30 to 45 seconds to log in, it takes 2 to 3 seconds.  

Much better!!

I would strongly recommend that the man page for sssd.conf be modified to point out that 'ignore_group_members = true' will greatly improve the speed of group lookups in pam_sss and 'id $user', and that it will also suppress listing user members in output from 'getent group $groupname'.

I think we can close this case.

Comment 5 Lukas Slebodnik 2014-12-10 19:40:38 UTC
(In reply to John Dickerson from comment #4)
> I would strongly recommend that the man page for sssd.conf be modified to
> point out that 'ignore_group_members = true' will greatly improve the speed
> of group lookups in pam_sss and 'id $user', and that it will also suppress
> listing user members in output from 'getent group $groupname'.

I suppose you are a native speaker. 

We will be very thankful if you could provide text which is better according to you. Contribution in any form is welcomed.

Comment 6 Jakub Hrozek 2014-12-10 20:44:41 UTC
(In reply to John Dickerson from comment #4)
> Thanks for the suggestions.
> 
> I was never sure what 'ignore_group_members' would do, but I think that is
> the solution.  If you set "ignore_group_members = true", then group lookups
> are much much faster in general.  
> 

This option makes all groups appear empty from the point of view of getgrnam(3) or getgrgid(3).

> The only downside is that 'getent group $somegroup' doesn't list the users
> in the group.  Not a big deal.  In fact, there's an argument to be made that
> it's better from a security point of view to suppress listing the group
> members.

Correct! In general, most security mechanisms only care about the other way around, mostly represented by getgrouplist() call, that is, what groups is the user a member of.

> 
> So now, instead of taking 30 to 45 seconds to log in, it takes 2 to 3
> seconds.  
> 
> Much better!!
> 
> I would strongly recommend that the man page for sssd.conf be modified to
> point out that 'ignore_group_members = true' will greatly improve the speed
> of group lookups in pam_sss and 'id $user', and that it will also suppress
> listing user members in output from 'getent group $groupname'.

I agree with Lukas that we'd appreciate clarification of the man page. If you can suggest better wording, I'm sure we can push it upstream.

Comment 7 Jakub Hrozek 2014-12-11 09:07:50 UTC
Since we found a workaround for the performance issue you were seeing, I'm going to close this bugzilla.

Feel free to add any suggestions for the man page here or send them to the  sssd-devel mailing list or even directly to one of the developers.

Thanks for filing the bug!

Comment 8 John Dickerson 2014-12-11 18:49:46 UTC
Here's a suggested change to the man page for sssd.conf:


       ignore_group_members (bool)
           Do not return group members for group lookups.

           If set to TRUE, suppresses the listing of group members in 'getent group $groupname'.  This can also make access provider checks for group membership significantly faster, especially for groups containing many members.  

           Default: FALSE

Comment 9 Jakub Hrozek 2014-12-12 09:41:11 UTC
I slightly reworded the first sentence to also describe what the underlying group calls are. Attached to this bugzilla is a proposed patch, does it look good to you?

Comment 10 Jakub Hrozek 2014-12-12 09:41:51 UTC
Created attachment 967534 [details]
A proposed man page patch.


Note You need to log in before you can comment on or make changes to this bug.