Bug 1474834

I believe the inconsistencies may have been an issue with a DC, results seem to be consistent now. I am still missing any IPA domain groups and Universal AD groups for trusted users on the client. I also saw this in the log, which I previously did not:

[sssd[be[ipa.us.int.kn]]] [get_groups_dns] (0x0400): Root domain uses fully-qualified names, objects might not be correctly added to groups with short names.

Hi,
I'm sorry this bug report was stuck for so long. Could you please elaborate on comment #2? Could you give me an example of a failing user lookup and describe in detail what information are you missing and what information do you expect to see instead?

If possible, please also include the sssd debug logs and config files.

Thank you.

Also, when you say a lookup is failing, is it failing only on an IPA client or also on the IPA server directly?

Hey Jakub,

No worries on the delay. The id lookup was failing on the client only. I made the following changes to the sssd config and it seems to be working, in the sense that I can see IPA domain groups but am still missing some AD groups (Which may not really be important)

I set the following timeouts on the client sssd.conf:

ldap_search_timeout = 600
ldap_opt_timeout = 600

This gave more consistency in displaying AD groups, but displayed no IPA groups.
[root@lxmatazan200s sssd]# id Matt.Agresta.adm.int.kn| tr "," "\n"
uid=835234833(matt.agresta.adm.int.kn) gid=835234833(matt.agresta.adm.int.kn) groups=835234833(matt.agresta.adm.int.kn)
835239512(amer.lex.admins.int.kn)
1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn)
835200512(domain admins.int.kn)
835234944(linux.adm.int.kn)
835236713(amer.mc4.admins.int.kn)
1407040638(ham_rz_ug kncetransfer users.int.kn)
835236712(amer.mc3.users.int.kn)
157001115(ad_forest_replicators.kn)

I then removed the ignore_group_members line on the ipa server:  

#subdomain_inherit = ignore_group_members
#ignore_group_members = True

Now its showing IPA groups but only a couple AD groups. COnsidering linux.adm.int.kn is an external member of engineering.all_external, which is a member of engineering.all, I understand why this resolves my issue. It also slows down the look ups considerably as we have a large AD environment. What I do not understand is why I am losing AD groups. This user is a direct member of Domain Admins and linux.adm.int.kn.

[root@lxmatazan200s sssd]# id Matt.Agresta.adm.int.kn| tr "," "\n"
uid=835234833(matt.agresta.adm.int.kn) gid=835234833(matt.agresta.adm.int.kn) groups=835234833(matt.agresta.adm.int.kn)
835239512(amer.lex.admins.int.kn)
835200512(domain admins.int.kn)
34000012(engineering.all)

On the server (throughout all of the changes described above) the lookup returns all groups:

[root@lxipaazan100s ~]# id Matt.Agresta.adm.int.kn | tr "," "\n"
uid=835234833(matt.agresta.adm.int.kn) gid=835234833(matt.agresta.adm.int.kn) groups=835234833(matt.agresta.adm.int.kn)
835234944(linux.adm.int.kn)
835200512(domain admins.int.kn)
835239512(amer.lex.admins.int.kn)
1407040638(ham_rz_ug kncetransfer users.int.kn)
835236712(amer.mc3.users.int.kn)
1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn)
157001115(ad_forest_replicators.kn)
835236713(amer.mc4.admins.int.kn)
835200513(adgroup)
835202855(dg-us-all.int.kn)
34000012(engineering.all)
34000013(usa_ad_users)


Here is the Client sssd.conf. Note that settings use_fully_qualified_names to True makes no difference in results, except showing the fqdn.

[domain/ipa.us.int.kn]

cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname = lxmatazan200s.us.int.kn
chpass_provider = ipa
ipa_server = lxipaazan100s.ipa.us.int.kn
dns_discovery_domain = ipa.us.int.kn

#Needed to fix bug preventing resolution of names with @ symbol in them
#Can be removed in 7.4
re_expression = (((?P<name>.+@.+)@(?P<domain>.+))|((?P<name>[^@]+)@?(?P<domain>[^@]*$)))
use_fully_qualified_names = False

ldap_search_timeout = 600
ldap_opt_timeout = 600

selinux_provider = none
debug_level = 9

[sssd]
debug_level = 9
services = nss, sudo, pam, ssh, pac
domains = ipa.us.int.kn
subdomain_enumerate = all

[nss]
homedir_substring = /home
debug_level = 9

I will add the logs of the latest lookup. I am ok with these results given that I only need to see the IPA groups for HBAC and Sudo rules, however lookups and logins are very slow. I will look into implementing the other options described in your performance tuning blog, but in the mean time is there anything I can do to ignore group members and see all groups? 

Thanks for all of your help.

Created attachment 1314733 [details]
/var/log/sssd/sssd.log

Created attachment 1314734 [details]
/var/log/sssd/sssd_ipa.us.int.kn.log

Any updates on this?

It seems to work ok without any of the performance tuning options. I was under the impression ignore_group_members would not break groups for AD users.

I have also run into an issue getting sssd back to a working state. After I revert back to a working sssd config I am unable to get it working again, the backend keeps going offline. I have tried restarting sssd service, using sss_cache to clean up the cache, manually removing the cache and even rebooting the servers, both for client and master. Despite all of this the client remains broken. Eventually it seems to come online but takes me hours to get it working again and I am never sure how or why its working.

Increasing the timeout on the client makes sense in a larger AD environment. Since the client gets all data from the IPA server the request to the server might need some time if the requested data is not already in the cache of the server.

The 'Root domain uses fully-qualified names...' warning is a result of using 'use_fully_qualified_names = True' on the client. Do you use it on the server, too? From the server-side id output I guess not, but can you nevertheless send the sssd.conf from the server as well?

About ignore_group_members, does the id command on the server show all expected groups if

    subdomain_inherit = ignore_group_members
    ignore_group_members = True

is enabled in sssd.conf on the server?

What is the scope (domain local, global or univerasal) of linux.adm.int.kn? Please note that only global and universal groups can be used in the IPA external groups.

The latest SSSD logs you have send show some issues looking up groups by GID on the client. Can you repeat this test, with ignore_group_members enabled on the server if id on the server still shows the expected results and after you called the id command on the server (to make sure all data is in the server's cache). Then send the SSSD logs from the client and the server covering the time of the id request. Most interesting would be the domain logs and the sssd_nss.log both with debug_level=10.

I was not using 'use_fully_qualified_names' on the server. I have removed it from the client, as well as the regex considering only a small amount of our AD users are affected by the @ symbol issue.

With the ignore_group_members set to true on the server, the server is still able to resolve all groups, but the client is not able to.

linux.adm.int.kn has a global scope

Server sssd.conf
[domain/ipa.us.int.kn]
cache_credentials = True
krb5_store_password_if_offline = True
krb5_validate = false
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = lxipaazan100s.ipa.us.int.kn
chpass_provider = ipa
ipa_server = lxipaazan100s.ipa.us.int.kn
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_server_mode = True

subdomains_provider = ipa
debug_level = 10

subdomain_inherit = ignore_group_members
ignore_group_members = True

[sssd]
domains = ipa.us.int.kn
config_file_version = 2
debug_level = 10
services = nss, sudo, pam, ssh, pac
subdomain_enumerate = all

[nss]
memcache_timeout = 600
homedir_substring = /home
debug_level = 10


Client sssd.conf
[domain/ipa.us.int.kn]

cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname = lxmatazan100s.us.int.kn
chpass_provider = ipa
ipa_server = _srv_, lxipaazan100s.ipa.us.int.kn
dns_discovery_domain = ipa.us.int.kn

selinux_provider = none
debug_level = 10

ldap_search_timeout = 600
ldap_opt_timeout = 600

[sssd]
services = nss, sudo, pam, ssh
debug_level = 10

domains = ipa.us.int.kn
[nss]
debug_level = 10
homedir_substring = /home

ID lookup on Server:
[root@lxipaazan100s sssd]# id Matt.Agresta.adm.int.kn
uid=835234833(matt.agresta.adm.int.kn) gid=835234833(matt.agresta.adm.int.kn) groups=835234833(matt.agresta.adm.int.kn),835234944(linux.adm.int.kn),835200512(domain admins.int.kn),835239512(amer.lex.admins.int.kn),1407040638(ham_rz_ug kncetransfer users.int.kn),835236712(amer.mc3.users.int.kn),1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn),157001115(ad_forest_replicators.kn),835236713(amer.mc4.admins.int.kn),835200513(adgroup),835202855(dg-us-all.int.kn),34000012(engineering.all),34000013(usa_ad_users)

ID Lookup on Client:
[root@lxmatazan100s sssd]# id Matt.Agresta.adm.int.kn
uid=835234833(matt.agresta.adm.int.kn) gid=835234833(matt.agresta.adm.int.kn) groups=835234833(matt.agresta.adm.int.kn),835239512(amer.lex.admins.int.kn),1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn),835200512(domain admins.int.kn),835234944(linux.adm.int.kn),835236713(amer.mc4.admins.int.kn),1407040638(ham_rz_ug kncetransfer users.int.kn),835236712(amer.mc3.users.int.kn),157001115(ad_forest_replicators.kn)


These logs where empty after running these lookups on both client and server, despite logging set to 10:
sssd_nss.log
sssd.log

Created attachment 1320587 [details]
Client log for id lookup

This lookup was performed on the server first and then on the client. The server has ignore_group_members enabled.

Created attachment 1320588 [details]
Server log for ID lookup

ID lookup on server with ignore_group_members enabled. The correct groups where returned.

The group 'adgroup' looks odd, from the GID I would expect that it is 'Domain Users.int.kn'. Do you know where this group name is coming from? Do you have by chance a group with that name and GID 835200513 in /etc/group?

The IPA Server does have a group with that name and GID in the /etc/group file, I am not sure how it got there, but I did not add it in.

[root@lxipaazan100s ~]# grep 835200513 /etc/group
adgroup:x:835200513:


I also did not set up the IPA groups usa_ad_users and usa_ad_users_external.

(In reply to Matt.Agresta from comment #15)
> The IPA Server does have a group with that name and GID in the /etc/group
> file, I am not sure how it got there, but I did not add it in.
> 
> [root@lxipaazan100s ~]# grep 835200513 /etc/group
> adgroup:x:835200513:
> 

Please remove this from /etc/group and any other group in /etc/group with the same name or GID as any IPA or AD group.

Then please check again if now the group memberships on the client are more reliable with 'ignore_group_members = True'.

> 
> I also did not set up the IPA groups usa_ad_users and usa_ad_users_external.

Removed the group from the /etc/group file and lookups are consistent now between client and server and everything seems to be running a lot faster. I am still not sure where this came from, I confirmed with my colleagues that no one manually added it in. The template we use is set up for direct integration with AD so I am guessing sssd automatically ads that group?


Thank you so much for identifying this, I have been looking at this issue for several months.

(In reply to Matt.Agresta from comment #17)
> Removed the group from the /etc/group file and lookups are consistent now
> between client and server and everything seems to be running a lot faster. I
> am still not sure where this came from, I confirmed with my colleagues that
> no one manually added it in. The template we use is set up for direct
> integration with AD so I am guessing sssd automatically ads that group?
> 

No, SSSD does not write to /etc/group and if SELinux is enable I think SSSD is not even allowed to write to the file.

Maybe the group was added to see if it helps to speed up lookups? Since the GID is the same as the 'Domain Users' group the lookup without 'ignore_group_members = True' might take some time because the group contains all AD users?

> 
> Thank you so much for identifying this, I have been looking at this issue
> for several months.

Glad I could help and it is working for you now. Do you think the ticket can be closed?

Sumit, you may close the ticket, have a nice day.

Thank you.