Bug 750388

Summary: Authconfig needs to handle new 'initgroups' nsswitch map (was: secondary groups not populated)
Product: [Fedora] Fedora Reporter: Andrew McNabb <amcnabb>
Component: authconfigAssignee: Tomas Mraz <tmraz>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16CC: awilliam, fweimer, jhrozek, k.georgiou, rdieter, sbose, sgallagh, ssorce, tmraz
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: https://fedoraproject.org/wiki/Common_F16_bugs#nsswitch-group-membership
Fixed In Version: authconfig-6.1.16-2.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 751450 (view as bug list) Environment:
Last Closed: 2011-11-25 23:23:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sssd_LDAP.log
none
sssd.conf
none
output from ldbsearch none

Description Andrew McNabb 2011-10-31 23:01:42 UTC
Description of problem:

In Fedora 16, users' secondary groups are not being populated correctly. If a user runs "groups", only the primary group appears.

Version-Release number of selected component (if applicable):

sssd-1.6.2-5.fc16.x86_64

How reproducible:

always

Steps to Reproduce:
1.
2.
3.
  
Actual results:

Only the first group appears.

Expected results:

Groups should list all groups, not just the first group.

Additional info:

The backend is LDAP. The groups work correctly in Fedora 15 with the exact same configuration files and using the exact same LDAP server.

What other information would be helpful? Thanks.

Comment 1 Jakub Hrozek 2011-11-01 09:19:36 UTC
(In reply to comment #0)
> What other information would be helpful? Thanks.

Debug logs. Please add "debug_level=10" to the domain section of sssd.conf, restart sssd and then attach /var/log/sssd/sssd_domain.log

Comment 2 Stephen Gallagher 2011-11-01 11:16:02 UTC
I'd also like to see your (sanitized) sssd.conf and a little information about your LDAP layout.

Comment 3 Andrew McNabb 2011-11-01 15:24:17 UTC
Created attachment 531151 [details]
sssd_LDAP.log

I'm attaching sssd_LDAP.log with debug_level set to 10. By the way, the groups "wheel" and "guest" are both groups that the "amcnabb" user should be a member of.

Comment 4 Andrew McNabb 2011-11-01 15:25:53 UTC
Created attachment 531152 [details]
sssd.conf

Comment 5 Andrew McNabb 2011-11-01 15:30:38 UTC
(In reply to comment #2)
> I'd also like to see your (sanitized) sssd.conf and a little information about
> your LDAP layout.

Our users are in ou=people,dc=aml,dc=cs,dc=byu,dc=edu and our groups are in ou=group,dc=aml,dc=cs,dc=byu,dc=edu. The wheel group is cn=wheel,ou=group,dc=aml,dc=cs,dc=byu,dc=edu.  The ldif for the wheel group is:

"""
dn: cn=wheel
cn: wheel
gidNumber: 9999
objectClass: posixGroup
objectClass: top
structuralObjectClass: posixGroup
memberUid: root
memberUid: amcnabb
""" (with some additional memberUid entries)

The "guest" group is much the same, except its gid is 10001.

I'm not sure if this is the sort of LDAP layout information that you're looking for, so feel free to ask any additional questions.

Comment 6 Stephen Gallagher 2011-11-01 15:37:59 UTC
Could you try setting 'enumerate = false', deleting/backing up your /var/lib/sss/db/cache_LDAP.ldb file and restarting SSSD?

I'd like to know if there might be a bug in the enumeration code that's causing your group memberships to disappear.

We mostly test with enumerate=false, because in most environments it's much faster (which is why it's the default)

Comment 7 Andrew McNabb 2011-11-01 15:44:12 UTC
Setting "enumerate=false" and deleting cache_LDAP.ldb doesn't seem to have made any difference.

(We only have about 25 users, so presumably enumerate=true shouldn't be slow.)

Comment 8 Stephen Gallagher 2011-11-01 15:52:50 UTC
What happens if you do:
'getent group wheel'

Do you get the correct set of users back?

Comment 9 Andrew McNabb 2011-11-01 15:58:01 UTC
Yes, "getent group wheel" gives the correct set of users.

Comment 10 Stephen Gallagher 2011-11-01 17:37:33 UTC
Andrew, what version of glibc do you have on the system?

rpm -q glibc

Comment 11 Andrew McNabb 2011-11-01 18:06:30 UTC
glibc-2.14.90-14.x86_64

Comment 12 Andrew McNabb 2011-11-03 15:46:36 UTC
I noticed today that PolicyKit successfully finds the members of the wheel group (polkit is configured with "AdminIdentities=unix-group:wheel"). It's interesting that both polkit and "getent group wheel" work.

Is there anything else that I should try to help track down the problem?

Comment 13 Stephen Gallagher 2011-11-04 16:24:00 UTC
(In reply to comment #12)
> I noticed today that PolicyKit successfully finds the members of the wheel
> group (polkit is configured with "AdminIdentities=unix-group:wheel"). It's
> interesting that both polkit and "getent group wheel" work.
> 
> Is there anything else that I should try to help track down the problem?

Does the 'wheel' group also exist in /etc/groups?

Comment 14 Andrew McNabb 2011-11-04 16:44:57 UTC
No--our postinstall script removes it to make sure that there's no conflict.

Comment 15 Andrew McNabb 2011-11-04 16:55:35 UTC
Created attachment 531809 [details]
output from ldbsearch

I saw an older bug where the reporter was asked to run ldbsearch. I don't know if the information is helpful here, but I'll attach it just in case. I generated the attachment by running the following:

# ldbsearch -H /var/lib/sss/db/cache_LDAP.ldb |grep -i -v pass >~amcnabb/tmp/ldbsearch.txt
asq: Unable to register control with rootdse!
#

The only interesting thing I noticed in the output was that in the record for amcnabb, it correctly included "memberof: name=guest,cn=groups,cn=LDAP,cn=sysdb" and "memberof: name=wheel,cn=groups,cn=LDAP,cn=sysdb".

Comment 16 Stephen Gallagher 2011-11-04 17:05:16 UTC
Andrew, what output do you get from 'id amcnabb'

I saw someone else reporting earlier today that 'groups' was missing groups for them too, but 'id username' had everything correct. If so, I think this is actually a glibc bug, not an SSSD bug.

Your LDB date looks perfect, so there's no reason it should not be reporting correctly.

Comment 17 Andrew McNabb 2011-11-04 17:08:40 UTC
Unfortunately, id is also showing incomplete information:

# id amcnabb
uid=10005(amcnabb) gid=10000(aml) groups=10000(aml)
#

Comment 18 Andrew McNabb 2011-11-04 17:13:40 UTC
I found it! It turns out that the following was in nsswitch.conf:

passwd:     files sss
shadow:     files sss
group:      files sss
initgroups: files

When I switched the last line to:

initgroups: files sss

it started working.

I'm thinking this might be an authconfig or anaconda bug.

Comment 19 Stephen Gallagher 2011-11-04 17:15:53 UTC
(In reply to comment #18)
> When I switched the last line to:
> 
> initgroups: files sss


What the heck? When did they add an 'initgroups' line? That's definitely new...

Yeah, if that's being added by default now, we need to have authconfig make sure to handle this.

If you remove that line completely (or comment it out) do things work again?

Comment 20 Andrew McNabb 2011-11-04 17:21:09 UTC
(In reply to comment #19)
> What the heck? When did they add an 'initgroups' line? That's definitely new...

That's what I thought. :)
 
> Yeah, if that's being added by default now, we need to have authconfig make
> sure to handle this.
> 
> If you remove that line completely (or comment it out) do things work again?

If I comment out the line, it works correctly.

Comment 21 Andrew McNabb 2011-11-04 17:24:31 UTC
So is this a glibc bug for including an unnecessary line that complicates life in the standard nsswitch.conf, or is it a authconfig bug for not writing the new initgroups line?

Also, this should probably be documented in the Fedora 16 Release Notes, right?

Comment 22 Stephen Gallagher 2011-11-04 17:26:25 UTC
Thanks for your patience and help in working this out. It looks like this is
something new that glibc added in F16 without advertising, and it's breaking
any non-files-based deployments on Fedora, because authconfig doesn't know to
update it.

I'm going to reassign this to authconfig.

Comment 23 Andrew McNabb 2011-11-04 17:30:24 UTC
I'm glad I could help. For now, I'm just removing the initgroups line in my postinstall because I can't think of any reason that it would be helpful to have initgroups be defined separately from groups.

Comment 24 Tomas Mraz 2011-11-04 18:02:28 UTC
To me it seems like default nsswitch.conf from glibc should have it simply commented out if in this case the value is copied from the groups value.

I can certainly add the patch to authconfig to support it - it will be trivial. However such a change without any heads up notice is really not nice from glibc.

Comment 25 Andrew McNabb 2011-11-04 18:17:46 UTC
(In reply to comment #24)
> I can certainly add the patch to authconfig to support it - it will be trivial.
> However such a change without any heads up notice is really not nice from
> glibc.

Indeed. Is there anyone in particular from glic that should be cc'ed on this bug?

Comment 26 Tomas Mraz 2011-11-04 18:49:27 UTC
Please clone this bug for glibc component.

Comment 27 Adam Williamson 2011-11-04 20:24:01 UTC
Yeah, we should definitely note this. sigh, glibc.

So, for a layman, what exactly is the impact of this bug? When using a non-file backend for user accounts, secondary groups don't work right? Fix is to edit nsswitch.conf ?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 28 Andrew McNabb 2011-11-04 20:46:54 UTC
(In reply to comment #27)
> Yeah, we should definitely note this. sigh, glibc.
> 
> So, for a layman, what exactly is the impact of this bug? When using a non-file
> backend for user accounts, secondary groups don't work right?

Exactly.

> Fix is to edit nsswitch.conf ?

The easiest fix is to remove the "initgroups" line from nsswitch.conf. It might be possible to fix by upgrading to a hypothetical new glibc RPM, but I'm not sure if this is feasible once authconfig has modified the file.

Comment 29 Simo Sorce 2011-11-04 21:30:40 UTC
Not only it is the easiest thing to remove the initgroups line.
It is also the correct thing to do.

The initgroups configuration option completely changes the behaviour of glibc wrt secondary groups. It is a clean break of what admins and user expect (whether the original behaviour was sane or not doesn't matter any more).

The init groups line should be commented out and a prominent comment guarding against enabling it unless you *really* know what you are doing should be added to nsswitch.conf file.

I am not sure how to unbreak existing installations, but I strongly believe extra work needs to be done to unbreak F16 machines with a 0 day update even if that means adding some nasty ugly post install scriplet.

Comment 30 Andrew McNabb 2011-11-04 21:43:48 UTC
(In reply to comment #29)
> The initgroups configuration option completely changes the behaviour of glibc
> wrt secondary groups. It is a clean break of what admins and user expect
> (whether the original behaviour was sane or not doesn't matter any more).

By the way, I couldn't find any documentation about how the behavior of glibc changes with this option. In fact, I opened bug #751429 earlier because this option is not listed in the nsswitch.conf(5) man page.

Comment 31 Andrew McNabb 2011-11-04 21:54:10 UTC
I guess this is informative:

"""Together with a previous patch which introduced the initgroups
entry in nsswitch.conf this patch allows more customization of
the lookups for initgroups/getgrouplist.  Nothing changes if
the groups entry in nsswitch.conf is used.  If the initgroups entry
is used instead the code now doesn't automatically continue looking
for more entries aftedr a successful lookup.  Instead the normal
rules are followed which do specify that by default no more
service is consulted.  This can be overwritten with
	[SUCCESS=continue]
appropriately placed in the line."""

http://sopc.et.ntust.edu.tw/?p=glibc.git;a=patch;h=7b3b0b2a63f7e980adb630550c0dc9639ec09d7f

That really does sound like a dangerous change in behavior.

Comment 32 Adam Williamson 2011-11-04 22:46:36 UTC
If we ship a 0-day update to fix this, then at least any *network* install of F16 should be safe, as it will have the fixed nsswitch.conf from day #1. It does sound like it might be a good idea to try and fix installed systems also, though it might be tricky.

can someone file a new bug for glibc and post the URL here? thanks.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 33 Tomas Mraz 2011-11-04 22:57:24 UTC
The glibc bug clone is bug 751450

Comment 34 Tomas Mraz 2011-11-04 22:59:36 UTC
And the authconfig build that handles the initgroups entry is here:
http://koji.fedoraproject.org/koji/buildinfo?buildID=272429

Note that authconfig will not add or uncomment the initgroups entry if it is not there. It will just add sssd or ldap to it if it is present.

Comment 35 Adam Williamson 2011-11-08 05:50:09 UTC
note for future reference: when I say "if we ship a zero-day update to fix this" it can be taken as a kind of hint that it would be nice for someone to submit a zero-day update.

the authconfig build referred to has not yet been _submitted_ as an update, much less accepted and pushed.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 36 Tomas Mraz 2011-11-08 07:32:39 UTC
I thought that the zero-day update would be in glibc.

Comment 37 Fedora Update System 2011-11-08 11:54:14 UTC
authconfig-6.1.16-2.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/authconfig-6.1.16-2.fc16

Comment 38 Andrew McNabb 2011-11-08 16:39:52 UTC
(In reply to comment #36)
> I thought that the zero-day update would be in glibc.

Agreed. It's nice that authconfig can deal with this, but glibc is where the time-critical fix really needs to happen.

Comment 39 Fedora Update System 2011-11-10 17:24:24 UTC
Package authconfig-6.1.16-2.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing authconfig-6.1.16-2.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-15563
then log in and leave karma (feedback).

Comment 40 Adam Williamson 2011-11-10 21:41:20 UTC
I don't really care who fixes what, I was just kind of expecting anyone who could fix anything would recognize the importance of doing so quickly...



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 41 Fedora Update System 2011-11-25 23:23:34 UTC
authconfig-6.1.16-2.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.