Bug 1984591

Summary:	After sssd update to 1.16.5-10.el7_9.8.x86_64 the customer is facing slow connection/authentication (due to discovery of unexpected AD domains)
Product:	Red Hat Enterprise Linux 7	Reporter:	cilmar <cilmar>
Component:	sssd	Assignee:	Sumit Bose <sbose>
Status:	CLOSED ERRATA	QA Contact:	Dan Lavu <dlavu>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	7.9	CC:	aboscatt, atikhono, dlavu, grajaiya, jhrozek, jreznik, lslebodn, mhernon, millard.matt, mknittel, mzidek, pbrezina, sbose, tscherf
Target Milestone:	rc	Keywords:	Triaged, ZStream
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	All
OS:	All
Whiteboard:	sync-to-jira review
Fixed In Version:	sssd-1.16.5-10.el7_9.11	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-23 17:17:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description cilmar@redhat.com 2021-07-21 17:43:33 UTC

Description of problem:
 - After sssd update to 1.16.5-10.el7_9.8.x86_64 the customer is facing slow connection/authentication.

BEFORE UPGRADE:
[root@HOST01 ~]# date; time id sssd
Tue Jun 29 13:41:22 EDT 2021
uid=217(sssd) gid=987(sssd) groups=987(sssd)
  real    1m30.030s
  user    0m0.004s
  sys     0m0.004s

AFTER DOWNGRADE
[root@HOST01 ~]# date; time id sssd
Tue Jun 29 13:47:17 EDT 2021
uid=217(sssd) gid=987(sssd) groups=987(sssd)
 real    0m7.300s
 user    0m0.003s
 sys     0m0.004s
=======================================================================
Version-Release number of selected component (if applicable):
BEFORE:
  1.16.5-10.el7_9.7.x86_64

AFTER UPGRADE:
  1.16.5-10.el7_9.8.x86_64 
=======================================================================

How reproducible:
 Upgrading sssd

Steps to Reproduce:
  1. Upgrading sssd 
  2. Authentication and fetch users/groups are slowing.
  3. Restarting sssd and clear cache does NOT help.
======================================================================
Actual results:
  Slow authentication and fetch users.

Expected results:
  Normal behaviour as using previous version:
=====================================================================

Additional info:
A - We figure out a workaround adding the parameter "ad_enabled_domains " on sssd.conf, however the customer said that it worked for more than 3 years without add the below line: 
   ad_enabled_domains = domain1.com, domain2.example.com, domain3.example.com, domain4.example.com.

B- After add "ad_enabled_domains", the sssd is working properly again with the new sssd release.

C- We also noticed the below difference after the update:
[root@zzzkctst011 ~]# sssctl domain-list
  domain1.com
  domain2.example.com
  domain3.example.com
  domain4.example.com.
[root@HOST01 ~]# rpm -q sssd
  sssd-1.16.5-10.el7_9.7.x86_64
  **NOTE: JUST 4 DOMAINS ARE FETCHED.

But after updating it to version 1.16.5-10.0.1.el7_9.8 we are seeing this domains.

[root@HOST01 ~]# sssctl domain-list
  domain1.com
  domain2.example.com
  domain3.example.com
  domain4.example.com.
  DOMAIN.NET
  DOMAIN.DMZ
  DOMAIN5.COM
  DOMAIN.LOCAL
  **NOTE: MANY MORE DOMAINS ARE FETCHED SOME OF THEM INATIVES.

[root@HOST01 ~]# rpm -q sssd
  sssd-1.16.5-10.el7_9.8.x86_64

D- I suspect the fixed #4980 is affecting this part:
   https://sssd.io/release-notes/sssd-1.16.5.html

Comment 5 Alexey Tikhonov 2021-07-21 18:34:58 UTC

(In reply to cilmar from comment #0)
> 
> C- We also noticed the below difference after the update:
> [root@zzzkctst011 ~]# sssctl domain-list
>   domain1.com
>   domain2.example.com
>   domain3.example.com
>   domain4.example.com.
> [root@HOST01 ~]# rpm -q sssd
>   sssd-1.16.5-10.el7_9.7.x86_64
>   **NOTE: JUST 4 DOMAINS ARE FETCHED.
> 
> But after updating it to version 1.16.5-10.0.1.el7_9.8 we are seeing this
> domains.
> 
> [root@HOST01 ~]# sssctl domain-list
>   domain1.com
>   domain2.example.com
>   domain3.example.com
>   domain4.example.com.
>   DOMAIN.NET
>   DOMAIN.DMZ
>   DOMAIN5.COM
>   DOMAIN.LOCAL
>   **NOTE: MANY MORE DOMAINS ARE FETCHED SOME OF THEM INATIVES.
> 
> [root@HOST01 ~]# rpm -q sssd
>   sssd-1.16.5-10.el7_9.8.x86_64
> 
> D- I suspect the fixed #4980 is affecting this part:
>    https://sssd.io/release-notes/sssd-1.16.5.html

I doubt.

Rather bz 1935685 / https://github.com/SSSD/sssd/issues/5528

Comment 7 Matt Millard 2021-07-26 17:52:17 UTC

I can confirm that we are also seeing this after the recent update of sssd on both RHEL7 and RHEL8 clients.  It is adding more domains than it should based on the configuration file. 2 before the update and 10 after with the exact same configuration and multiple of these new ones are international with slower connections.  Seeing a long listing of a directory take over 2+ minutes.

Comment 9 Sumit Bose 2021-08-02 15:01:00 UTC

Hi,

I agree with Alexey that the fix for bz 1935685 made SSSD see more domains than before. And imo this is really a bug-fix because the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.". So SSSD's AD provider should by default try to discover all trusted domains in the AD forest.

If only a sub-set of the domains in the forest should be used by SSSD it is recommended to add 'ad_enabled_domains' with the list of domains which should be used. Please note that instead of adding this option to the 'sssd.conf' file directly it can be added by a config snippet, e.g. /etc/sssd/conf.d/restrict_domains.conf containing

    [domain/name.of.joined.domain]
    ad_enabled_domains = dom1.example.com, dom2.example.com, dom3.example.com

This file can even be added before the system is joined so that SSSD can pick it up during the first start. As 'sssd.conf' the snippet must be owned by root with 0600 permissions.

HTH

bye,
Sumit

Comment 15 Sumit Bose 2021-10-08 16:27:37 UTC

Upstream ticket:
https://github.com/SSSD/sssd/issues/5819

Comment 18 Alexey Tikhonov 2021-10-29 09:48:53 UTC

Upstream PR: https://github.com/SSSD/sssd/pull/5850

Comment 21 Alexey Tikhonov 2021-11-02 12:13:08 UTC

Pushed PR: https://github.com/SSSD/sssd/pull/5850

* `master`
    * 4c48c4a7792961cf8a228c76975ac370d32904e1 - ad: filter trusted domains
* `sssd-1-16`
    * 87aaf96ab7bd39698c41625d56602ca3de943b87 - ad: filter trusted domains

Comment 25 Dan Lavu 2021-11-11 05:12:24 UTC

Verified with an upgrade to sssd-1.16.5-10.el7_9.11.x86_64 from sssd-1.16.5-10.el7_9.10.x86_64


[root@ci-vm-10-0-103-180 yum.repos.d]# realm join domain-zf0b.com
Password for Administrator: 

[root@ci-vm-10-0-103-180 yum.repos.d]# service sssd restart
Redirecting to /bin/systemctl restart sssd.service

[root@ci-vm-10-0-103-180 yum.repos.d]# rpm -qa |grep sssd-1
sssd-1.16.5-10.el7_9.10.x86_64

[root@ci-vm-10-0-103-180 yum.repos.d]#  id administrator
uid=459000500(administrator) gid=459000513(domain users) groups=459000513(domain users),459000520(group policy creator owners),459000572(denied rodc password replication group),459000518(schema admins),459000519(enterprise admins),459000512(domain admins)


[root@ci-vm-10-0-103-180 yum.repos.d]# yum update -y
Loaded plugins: search-disabled-repos
rhel-Server                                                                                                                                                    | 2.8 kB  00:00:00     
rhel-Server-old                                                                                                                                                | 2.8 kB  00:00:00     
rhel-Server-optional                                                                                                                                           | 2.3 kB  00:00:00     
rhel-Server-optional-old                                                                                                                                       | 2.3 kB  00:00:00     
rhel-Server-optional/x86_64/primary                                                                                                                            | 995 kB  00:00:00     
rhel-Server-optional                                                                                                                                                        4688/4688
Resolving Dependencies
--> Running transaction check
---> Package libipa_hbac.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package libipa_hbac.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package libsss_autofs.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package libsss_autofs.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package libsss_certmap.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package libsss_certmap.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package libsss_idmap.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package libsss_idmap.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package libsss_nss_idmap.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package libsss_nss_idmap.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package libsss_sudo.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package libsss_sudo.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package lshw.x86_64 0:B.02.18-17.el7 will be updated
---> Package lshw.x86_64 0:B.02.19-0.1.20180614git028f6b2.beaker.1.el7bkr.1 will be an update
---> Package python-sssdconfig.noarch 0:1.16.5-10.el7_9.10 will be updated
---> Package python-sssdconfig.noarch 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-ad.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-ad.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-client.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-client.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-common.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-common.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-common-pac.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-common-pac.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-ipa.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-ipa.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-krb5.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-krb5.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-krb5-common.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-krb5-common.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-ldap.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-ldap.x86_64 0:1.16.5-10.el7_9.11 will be an update
---> Package sssd-proxy.x86_64 0:1.16.5-10.el7_9.10 will be updated
---> Package sssd-proxy.x86_64 0:1.16.5-10.el7_9.11 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================
 Package                                 Arch                         Version                                                                 Repository                         Size
======================================================================================================================================================================================
Updating:
 libipa_hbac                             x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       157 k
 libsss_autofs                           x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       159 k
 libsss_certmap                          x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       190 k
 libsss_idmap                            x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       162 k
 libsss_nss_idmap                        x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       168 k
 libsss_sudo                             x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       158 k
 lshw                                    x86_64                       B.02.19-0.1.20180614git028f6b2.beaker.1.el7bkr.1                        beaker-el7                        361 k
 python-sssdconfig                       noarch                       1.16.5-10.el7_9.11                                                      rhel-Server                       181 k
 sssd                                    x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       149 k
 sssd-ad                                 x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       304 k
 sssd-client                             x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       229 k
 sssd-common                             x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       1.5 M
 sssd-common-pac                         x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       223 k
 sssd-ipa                                x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       385 k
 sssd-krb5                               x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       191 k
 sssd-krb5-common                        x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       225 k
 sssd-ldap                               x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       285 k
 sssd-proxy                              x86_64                       1.16.5-10.el7_9.11                                                      rhel-Server                       185 k

Transaction Summary
======================================================================================================================================================================================
Upgrade  18 Packages


-------------- SNIP ----------------

  Verifying  : sssd-krb5-common-1.16.5-10.el7_9.11.x86_64 1/36 
  Verifying  : sssd-client-1.16.5-10.el7_9.11.x86_64     2/36 
  Verifying  : libipa_hbac-1.16.5-10.el7_9.11.x86_64          3/36 
  Verifying  : sssd-proxy-1.16.5-10.el7_9.11.x86_64               4/36 
  Verifying  : sssd-ipa-1.16.5-10.el7_9.11.x86_64                 5/36 
  Verifying  : sssd-krb5-1.16.5-10.el7_9.11.x86_64                6/36 
  Verifying  : libsss_autofs-1.16.5-10.el7_9.11.x86_64            7/36 
  Verifying  : sssd-common-pac-1.16.5-10.el7_9.11.x86_64          8/36 
  Verifying  : libsss_nss_idmap-1.16.5-10.el7_9.11.x86_64         9/36 
  Verifying  : sssd-ldap-1.16.5-10.el7_9.11.x86_64    10/36 
  Verifying  : lshw-B.02.19-0.1.20180614git028f6b2.beaker.1.el7bkr.1.x86_64  11/36 
  Verifying  : libsss_idmap-1.16.5-10.el7_9.11.x86_64 12/36 
  Verifying  : sssd-1.16.5-10.el7_9.11.x86_64         13/36 
  Verifying  : sssd-common-1.16.5-10.el7_9.11.x86_64      14/36 
  Verifying  : sssd-ad-1.16.5-10.el7_9.11.x86_64            15/36 
  Verifying  : libsss_certmap-1.16.5-10.el7_9.11.x86_64     16/36 
  Verifying  : libsss_sudo-1.16.5-10.el7_9.11.x86_64        17/36 
  Verifying  : python-sssdconfig-1.16.5-10.el7_9.11.noarch  18/36 
  Verifying  : sssd-common-1.16.5-10.el7_9.10.x86_64      19/36 
  Verifying  : libsss_certmap-1.16.5-10.el7_9.10.x86_64   20/36 
  Verifying  : sssd-ad-1.16.5-10.el7_9.10.x86_64       21/36 
  Verifying  : libsss_sudo-1.16.5-10.el7_9.10.x86_64   22/36 
  Verifying  : libipa_hbac-1.16.5-10.el7_9.10.x86_64      23/36 
  Verifying  : python-sssdconfig-1.16.5-10.el7_9.10.noarch24/36 
  Verifying  : sssd-krb5-common-1.16.5-10.el7_9.10.x86_6425/36 
  Verifying  : lshw-B.02.18-17.el7.x86_64             26/36 
  Verifying  : sssd-krb5-1.16.5-10.el7_9.10.x86_64     27/36 
  Verifying  : sssd-client-1.16.5-10.el7_9.10.x86_64  28/36 
  Verifying  : sssd-proxy-1.16.5-10.el7_9.10.x86_64   29/36 
  Verifying  : libsss_nss_idmap-1.16.5-10.el7_9.10.x86_64   30/36 
  Verifying  : sssd-ipa-1.16.5-10.el7_9.10.x86_64          31/36 
  Verifying  : libsss_autofs-1.16.5-10.el7_9.10.x86_64     32/36 
  Verifying  : sssd-1.16.5-10.el7_9.10.x86_64              33/36 
  Verifying  : sssd-ldap-1.16.5-10.el7_9.10.x86_64         34/36 
  Verifying  : sssd-common-pac-1.16.5-10.el7_9.10.x86_64   35/36 
  Verifying  : libsss_idmap-1.16.5-10.el7_9.10.x86_64      36/36 

Updated:
  libipa_hbac.x86_64 0:1.16.5-10.el7_9.11                                 libsss_autofs.x86_64 0:1.16.5-10.el7_9.11              libsss_certmap.x86_64 0:1.16.5-10.el7_9.11         
  libsss_idmap.x86_64 0:1.16.5-10.el7_9.11                                libsss_nss_idmap.x86_64 0:1.16.5-10.el7_9.11           libsss_sudo.x86_64 0:1.16.5-10.el7_9.11            
  lshw.x86_64 0:B.02.19-0.1.20180614git028f6b2.beaker.1.el7bkr.1          python-sssdconfig.noarch 0:1.16.5-10.el7_9.11          sssd.x86_64 0:1.16.5-10.el7_9.11                   
  sssd-ad.x86_64 0:1.16.5-10.el7_9.11                                     sssd-client.x86_64 0:1.16.5-10.el7_9.11                sssd-common.x86_64 0:1.16.5-10.el7_9.11            
  sssd-common-pac.x86_64 0:1.16.5-10.el7_9.11                             sssd-ipa.x86_64 0:1.16.5-10.el7_9.11                   sssd-krb5.x86_64 0:1.16.5-10.el7_9.11              
  sssd-krb5-common.x86_64 0:1.16.5-10.el7_9.11                            sssd-ldap.x86_64 0:1.16.5-10.el7_9.11                  sssd-proxy.x86_64 0:1.16.5-10.el7_9.11             

Complete!
[root@ci-vm-10-0-103-180 yum.repos.d]# 

[root@ci-vm-10-0-103-180 yum.repos.d]#  id administrator
uid=459000500(administrator) gid=459000513(domain users) groups=459000513(domain users),459000520(group policy creator owners),459000572(denied rodc password replication group),459000518(schema admins),459000519(enterprise admins),459000512(domain admins)

[root@ci-vm-10-0-103-180 yum.repos.d]#  sssctl domain-list
domain-zf0b.com
child-zf0b.domain-zf0b.com
tdomain-zf0b.com

[root@ci-vm-10-0-103-180 yum.repos.d]# date; time id sssd
Thu Nov 11 00:10:59 EST 2021
uid=388(sssd) gid=387(sssd) groups=387(sssd)

real	0m0.006s
user	0m0.000s
sys	0m0.006s

[root@ci-vm-10-0-103-180 yum.repos.d]# rpm -qa | grep sssd-1
sssd-1.16.5-10.el7_9.11.x86_64

Comment 31 errata-xmlrpc 2021-11-23 17:17:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4793