Bug 1474834 - IPA client inconsistencies with AD trusted user's groups
IPA client inconsistencies with AD trusted user's groups
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sssd (Show other bugs)
7.3
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Jakub Hrozek
sssd-qe
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-25 09:27 EDT by Matt.Agresta@kuehne-nagel.com
Modified: 2017-09-06 08:00 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-06 08:00:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
client sssd domain logs (1.12 MB, text/plain)
2017-07-25 09:27 EDT, Matt.Agresta@kuehne-nagel.com
no flags Details
/var/log/sssd/sssd.log (88 bytes, text/plain)
2017-08-17 09:45 EDT, Matt.Agresta@kuehne-nagel.com
no flags Details
/var/log/sssd/sssd_ipa.us.int.kn.log (462.95 KB, text/plain)
2017-08-17 09:46 EDT, Matt.Agresta@kuehne-nagel.com
no flags Details
Client log for id lookup (462.95 KB, text/plain)
2017-08-31 09:10 EDT, Matt.Agresta@kuehne-nagel.com
no flags Details
Server log for ID lookup (1.88 MB, text/plain)
2017-08-31 09:11 EDT, Matt.Agresta@kuehne-nagel.com
no flags Details

  None (edit)
Description Matt.Agresta@kuehne-nagel.com 2017-07-25 09:27:49 EDT
Created attachment 1304212 [details]
client sssd domain logs

Description of problem:
Identity lookup of AD trusted user returns various results. The first lookup will return 'no such user', the second will return the user with only their primary group, any lookup after that will return a portion of secondary groups, but no IPA domain groups. User shown is Domain Admin, same results if user is none admin, user has no groups with @ symbol in name.

Version-Release number of selected component (if applicable):
[root@lxmatazan200s sssd]# ipa --version
VERSION: 4.4.0, API_VERSION: 2.213
[root@lxmatazan200s sssd]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)


How reproducible:
Every attempt results in the same

Steps to Reproduce:
1. Enroll machine into IPA domain
   ipa-client-install -U --domain ipa.us.int.kn --force-join --mkhomedir -p admin -w <PASSWORD>
2. Disable se_linux in sssd.conf
   selinux_provider = none
3. Restart sssd, clear the cache
   systemctl stop sssd.service; rm -rf /var/lib/sss/{db,mc}/*;systemctl start sssd.service;
4. Run identity lookup of AD user
   id matt.agresta.adm@usa.win.int.kn

Actual results:
[root@lxmatazan200s sssd]# id matt.agresta.adm@usa.win.int.kn
id: matt.agresta.adm@usa.win.int.kn: no such user
[root@lxmatazan200s sssd]# systemctl stop sssd.service; rm -rf /var/lib/sss/{db,mc}/*;systemctl start sssd.service;
[root@lxmatazan200s sssd]# id matt.agresta.adm@usa.win.int.kn
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn)
[root@lxmatazan200s sssd]# systemctl stop sssd.service; rm -rf /var/lib/sss/{db,mc}/*;systemctl start sssd.service;
[root@lxmatazan200s sssd]# id matt.agresta.adm@usa.win.int.kn
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn),1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn),835200512(domain admins@usa.win.int.kn),835236713(amer.mc4.admins@usa.win.int.kn),1407040638(ham_rz_ug kncetransfer users@ger.win.int.kn),835236712(amer.mc3.users@usa.win.int.kn),157001115(ad_forest_replicators@win.int.kn)

Expected results:
[root@lxipaazan100s slapd-IPA-US-INT-KN]# id Matt.Agresta.adm@usa.win.int.kn
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn),835234944(linux.adm@usa.win.int.kn),835200512(domain admins@usa.win.int.kn),1407040638(ham_rz_ug kncetransfer users@ger.win.int.kn),835236712(amer.mc3.users@usa.win.int.kn),1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn),157001115(ad_forest_replicators@win.int.kn),835236713(amer.mc4.admins@usa.win.int.kn),835200513(adgroup),835202855(dg-us-all@usa.win.int.kn),34100502(engineering.all)


Additional info:
sssd.conf
[domain/ipa.us.int.kn]
cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname = lxmatazan200s.us.int.kn
chpass_provider = ipa
ipa_server = lxipaazan100s.ipa.us.int.kn
dns_discovery_domain = ipa.us.int.kn
debug_level = 9

[sssd]
debug_level = 9
services = nss, sudo, pam, ssh, pac
subdomain_enumerate = all
domains = ipa.us.int.kn

[nss]
homedir_substring = /home
debug_level = 9
Comment 2 Matt.Agresta@kuehne-nagel.com 2017-08-03 16:45:00 EDT
I believe the inconsistencies may have been an issue with a DC, results seem to be consistent now. I am still missing any IPA domain groups and Universal AD groups for trusted users on the client. I also saw this in the log, which I previously did not:

[sssd[be[ipa.us.int.kn]]] [get_groups_dns] (0x0400): Root domain uses fully-qualified names, objects might not be correctly added to groups with short names.
Comment 3 Jakub Hrozek 2017-08-16 16:07:32 EDT
Hi,
I'm sorry this bug report was stuck for so long. Could you please elaborate on comment #2? Could you give me an example of a failing user lookup and describe in detail what information are you missing and what information do you expect to see instead?

If possible, please also include the sssd debug logs and config files.

Thank you.
Comment 4 Jakub Hrozek 2017-08-16 16:08:23 EDT
Also, when you say a lookup is failing, is it failing only on an IPA client or also on the IPA server directly?
Comment 5 Matt.Agresta@kuehne-nagel.com 2017-08-17 09:44:25 EDT
Hey Jakub,

No worries on the delay. The id lookup was failing on the client only. I made the following changes to the sssd config and it seems to be working, in the sense that I can see IPA domain groups but am still missing some AD groups (Which may not really be important)

I set the following timeouts on the client sssd.conf:

ldap_search_timeout = 600
ldap_opt_timeout = 600

This gave more consistency in displaying AD groups, but displayed no IPA groups.
[root@lxmatazan200s sssd]# id Matt.Agresta.adm@usa.win.int.kn| tr "," "\n"
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn)
835239512(amer.lex.admins@usa.win.int.kn)
1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn)
835200512(domain admins@usa.win.int.kn)
835234944(linux.adm@usa.win.int.kn)
835236713(amer.mc4.admins@usa.win.int.kn)
1407040638(ham_rz_ug kncetransfer users@ger.win.int.kn)
835236712(amer.mc3.users@usa.win.int.kn)
157001115(ad_forest_replicators@win.int.kn)

I then removed the ignore_group_members line on the ipa server:  

#subdomain_inherit = ignore_group_members
#ignore_group_members = True

Now its showing IPA groups but only a couple AD groups. COnsidering linux.adm@usa.win.int.kn is an external member of engineering.all_external, which is a member of engineering.all, I understand why this resolves my issue. It also slows down the look ups considerably as we have a large AD environment. What I do not understand is why I am losing AD groups. This user is a direct member of Domain Admins and linux.adm@usa.win.int.kn.

[root@lxmatazan200s sssd]# id Matt.Agresta.adm@usa.win.int.kn| tr "," "\n"
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn)
835239512(amer.lex.admins@usa.win.int.kn)
835200512(domain admins@usa.win.int.kn)
34000012(engineering.all)

On the server (throughout all of the changes described above) the lookup returns all groups:

[root@lxipaazan100s ~]# id Matt.Agresta.adm@usa.win.int.kn | tr "," "\n"
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn)
835234944(linux.adm@usa.win.int.kn)
835200512(domain admins@usa.win.int.kn)
835239512(amer.lex.admins@usa.win.int.kn)
1407040638(ham_rz_ug kncetransfer users@ger.win.int.kn)
835236712(amer.mc3.users@usa.win.int.kn)
1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn)
157001115(ad_forest_replicators@win.int.kn)
835236713(amer.mc4.admins@usa.win.int.kn)
835200513(adgroup)
835202855(dg-us-all@usa.win.int.kn)
34000012(engineering.all)
34000013(usa_ad_users)


Here is the Client sssd.conf. Note that settings use_fully_qualified_names to True makes no difference in results, except showing the fqdn.

[domain/ipa.us.int.kn]

cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname = lxmatazan200s.us.int.kn
chpass_provider = ipa
ipa_server = lxipaazan100s.ipa.us.int.kn
dns_discovery_domain = ipa.us.int.kn

#Needed to fix bug preventing resolution of names with @ symbol in them
#Can be removed in 7.4
re_expression = (((?P<name>.+@.+)@(?P<domain>.+))|((?P<name>[^@]+)@?(?P<domain>[^@]*$)))
use_fully_qualified_names = False

ldap_search_timeout = 600
ldap_opt_timeout = 600

selinux_provider = none
debug_level = 9

[sssd]
debug_level = 9
services = nss, sudo, pam, ssh, pac
domains = ipa.us.int.kn
subdomain_enumerate = all

[nss]
homedir_substring = /home
debug_level = 9

I will add the logs of the latest lookup. I am ok with these results given that I only need to see the IPA groups for HBAC and Sudo rules, however lookups and logins are very slow. I will look into implementing the other options described in your performance tuning blog, but in the mean time is there anything I can do to ignore group members and see all groups? 

Thanks for all of your help.
Comment 6 Matt.Agresta@kuehne-nagel.com 2017-08-17 09:45 EDT
Created attachment 1314733 [details]
/var/log/sssd/sssd.log
Comment 7 Matt.Agresta@kuehne-nagel.com 2017-08-17 09:46 EDT
Created attachment 1314734 [details]
/var/log/sssd/sssd_ipa.us.int.kn.log
Comment 8 Matt.Agresta@kuehne-nagel.com 2017-08-28 14:20:11 EDT
Any updates on this?

It seems to work ok without any of the performance tuning options. I was under the impression ignore_group_members would not break groups for AD users.

I have also run into an issue getting sssd back to a working state. After I revert back to a working sssd config I am unable to get it working again, the backend keeps going offline. I have tried restarting sssd service, using sss_cache to clean up the cache, manually removing the cache and even rebooting the servers, both for client and master. Despite all of this the client remains broken. Eventually it seems to come online but takes me hours to get it working again and I am never sure how or why its working.
Comment 10 Sumit Bose 2017-08-30 12:03:45 EDT
Increasing the timeout on the client makes sense in a larger AD environment. Since the client gets all data from the IPA server the request to the server might need some time if the requested data is not already in the cache of the server.

The 'Root domain uses fully-qualified names...' warning is a result of using 'use_fully_qualified_names = True' on the client. Do you use it on the server, too? From the server-side id output I guess not, but can you nevertheless send the sssd.conf from the server as well?

About ignore_group_members, does the id command on the server show all expected groups if

    subdomain_inherit = ignore_group_members
    ignore_group_members = True

is enabled in sssd.conf on the server?

What is the scope (domain local, global or univerasal) of linux.adm@usa.win.int.kn? Please note that only global and universal groups can be used in the IPA external groups.

The latest SSSD logs you have send show some issues looking up groups by GID on the client. Can you repeat this test, with ignore_group_members enabled on the server if id on the server still shows the expected results and after you called the id command on the server (to make sure all data is in the server's cache). Then send the SSSD logs from the client and the server covering the time of the id request. Most interesting would be the domain logs and the sssd_nss.log both with debug_level=10.
Comment 11 Matt.Agresta@kuehne-nagel.com 2017-08-31 09:08:38 EDT
I was not using 'use_fully_qualified_names' on the server. I have removed it from the client, as well as the regex considering only a small amount of our AD users are affected by the @ symbol issue.

With the ignore_group_members set to true on the server, the server is still able to resolve all groups, but the client is not able to.

linux.adm@usa.win.int.kn has a global scope

Server sssd.conf
[domain/ipa.us.int.kn]
cache_credentials = True
krb5_store_password_if_offline = True
krb5_validate = false
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = lxipaazan100s.ipa.us.int.kn
chpass_provider = ipa
ipa_server = lxipaazan100s.ipa.us.int.kn
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_server_mode = True

subdomains_provider = ipa
debug_level = 10

subdomain_inherit = ignore_group_members
ignore_group_members = True

[sssd]
domains = ipa.us.int.kn
config_file_version = 2
debug_level = 10
services = nss, sudo, pam, ssh, pac
subdomain_enumerate = all

[nss]
memcache_timeout = 600
homedir_substring = /home
debug_level = 10


Client sssd.conf
[domain/ipa.us.int.kn]

cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.us.int.kn
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname = lxmatazan100s.us.int.kn
chpass_provider = ipa
ipa_server = _srv_, lxipaazan100s.ipa.us.int.kn
dns_discovery_domain = ipa.us.int.kn

selinux_provider = none
debug_level = 10

ldap_search_timeout = 600
ldap_opt_timeout = 600

[sssd]
services = nss, sudo, pam, ssh
debug_level = 10

domains = ipa.us.int.kn
[nss]
debug_level = 10
homedir_substring = /home

ID lookup on Server:
[root@lxipaazan100s sssd]# id Matt.Agresta.adm@usa.win.int.kn
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn),835234944(linux.adm@usa.win.int.kn),835200512(domain admins@usa.win.int.kn),835239512(amer.lex.admins@usa.win.int.kn),1407040638(ham_rz_ug kncetransfer users@ger.win.int.kn),835236712(amer.mc3.users@usa.win.int.kn),1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn),157001115(ad_forest_replicators@win.int.kn),835236713(amer.mc4.admins@usa.win.int.kn),835200513(adgroup),835202855(dg-us-all@usa.win.int.kn),34000012(engineering.all),34000013(usa_ad_users)

ID Lookup on Client:
[root@lxmatazan100s sssd]# id Matt.Agresta.adm@usa.win.int.kn
uid=835234833(matt.agresta.adm@usa.win.int.kn) gid=835234833(matt.agresta.adm@usa.win.int.kn) groups=835234833(matt.agresta.adm@usa.win.int.kn),835239512(amer.lex.admins@usa.win.int.kn),1407040772(ham_rz_ug serverinstallation (lesen)@ger.win.int.kn),835200512(domain admins@usa.win.int.kn),835234944(linux.adm@usa.win.int.kn),835236713(amer.mc4.admins@usa.win.int.kn),1407040638(ham_rz_ug kncetransfer users@ger.win.int.kn),835236712(amer.mc3.users@usa.win.int.kn),157001115(ad_forest_replicators@win.int.kn)


These logs where empty after running these lookups on both client and server, despite logging set to 10:
sssd_nss.log
sssd.log
Comment 12 Matt.Agresta@kuehne-nagel.com 2017-08-31 09:10 EDT
Created attachment 1320587 [details]
Client log for id lookup

This lookup was performed on the server first and then on the client. The server has ignore_group_members enabled.
Comment 13 Matt.Agresta@kuehne-nagel.com 2017-08-31 09:11 EDT
Created attachment 1320588 [details]
Server log for ID lookup

ID lookup on server with ignore_group_members enabled. The correct groups where returned.
Comment 14 Sumit Bose 2017-09-01 15:41:08 EDT
The group 'adgroup' looks odd, from the GID I would expect that it is 'Domain Users@usa.win.int.kn'. Do you know where this group name is coming from? Do you have by chance a group with that name and GID 835200513 in /etc/group?
Comment 15 Matt.Agresta@kuehne-nagel.com 2017-09-05 07:54:21 EDT
The IPA Server does have a group with that name and GID in the /etc/group file, I am not sure how it got there, but I did not add it in.

[root@lxipaazan100s ~]# grep 835200513 /etc/group
adgroup:x:835200513:


I also did not set up the IPA groups usa_ad_users and usa_ad_users_external.
Comment 16 Sumit Bose 2017-09-05 08:47:39 EDT
(In reply to Matt.Agresta@kuehne-nagel.com from comment #15)
> The IPA Server does have a group with that name and GID in the /etc/group
> file, I am not sure how it got there, but I did not add it in.
> 
> [root@lxipaazan100s ~]# grep 835200513 /etc/group
> adgroup:x:835200513:
> 

Please remove this from /etc/group and any other group in /etc/group with the same name or GID as any IPA or AD group.

Then please check again if now the group memberships on the client are more reliable with 'ignore_group_members = True'.

> 
> I also did not set up the IPA groups usa_ad_users and usa_ad_users_external.
Comment 17 Matt.Agresta@kuehne-nagel.com 2017-09-05 10:06:51 EDT
Removed the group from the /etc/group file and lookups are consistent now between client and server and everything seems to be running a lot faster. I am still not sure where this came from, I confirmed with my colleagues that no one manually added it in. The template we use is set up for direct integration with AD so I am guessing sssd automatically ads that group?


Thank you so much for identifying this, I have been looking at this issue for several months.
Comment 18 Sumit Bose 2017-09-06 02:39:54 EDT
(In reply to Matt.Agresta@kuehne-nagel.com from comment #17)
> Removed the group from the /etc/group file and lookups are consistent now
> between client and server and everything seems to be running a lot faster. I
> am still not sure where this came from, I confirmed with my colleagues that
> no one manually added it in. The template we use is set up for direct
> integration with AD so I am guessing sssd automatically ads that group?
> 

No, SSSD does not write to /etc/group and if SELinux is enable I think SSSD is not even allowed to write to the file.

Maybe the group was added to see if it helps to speed up lookups? Since the GID is the same as the 'Domain Users' group the lookup without 'ignore_group_members = True' might take some time because the group contains all AD users?

> 
> Thank you so much for identifying this, I have been looking at this issue
> for several months.

Glad I could help and it is working for you now. Do you think the ticket can be closed?
Comment 19 Matt.Agresta@kuehne-nagel.com 2017-09-06 07:32:44 EDT
Sumit, you may close the ticket, have a nice day.
Comment 20 Sumit Bose 2017-09-06 08:00:50 EDT
Thank you.

Note You need to log in before you can comment on or make changes to this bug.