Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1430415

Summary: ldap_purge_cache_timeout in RHEL7.3 invalidate most of the entries once the cleanup task kicks in
Product: Red Hat Enterprise Linux 7 Reporter: Ming Davies <minyu>
Component: sssdAssignee: SSSD Maintainers <sssd-maint>
Status: CLOSED ERRATA QA Contact: shridhar <sgadekar>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: apeddire, dmoessne, fidencio, grajaiya, jhrozek, lslebodn, minyu, mkosek, mzidek, pbrezina, sgoveas, sssd-maint, striker, tscherf
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sssd-1.15.2-47.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1471808 (view as bug list) Environment:
Last Closed: 2017-08-01 09:04:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1420851, 1452916, 1471808    
Attachments:
Description Flags
latest sssd debug none

Description Ming Davies 2017-03-08 14:51:11 UTC 
Description of problem:
Customer said:"Our intention is to have the entire ldap catalog in the local cache and for that purpose we have set "enumerate = true" but but every 3 hours when ldap_purge_cache_timeout is done all accounts, groups and sudo rules is removed from the local cache. After 3 hours again all entries is back in the local cache". The customer is fully aware of the performance impact when using enumerate.

The customer claims that the issue is perceived on servers running 7.3 and sssd 1.14.0 release 43.el7_3.11, but seemed to work fine in rhel 6 and sssd 1.13.3 rel 22.


According to "man sssd-ldap", if enumeration is enabled, the cleanup task, i.e. ldap_purge_cache_timeout is required in order to detect entries removed from the server and can't be disabled!



Version-Release number of selected component (if applicable):
sssd 1.14.0 release 43.el7_3.11

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 9 Ming Davies 2017-04-11 13:43:55 UTC 
Created attachment 1270783 [details]
latest sssd debug
Comment 11 Pavel Březina 2017-04-12 10:01:36 UTC 
Hi, thank you for the logs. This is indeed an issue due to introduction of the timestamp cache to speed up performance.

1. Full enumeration fills cache with users and groups
 -- for next three hours enumeration will be usen entryUSN to fetch only new entries
2. Purge cache timeout kicks in
 -- Full enumeration is done, but it only updates timestamp cache
 -- We search users in data cache with expiriration time filter
 -- We end up deleting all users
3. Again only smart enumeration with entryUSN is used, which won't get any result
4. Purge cache timeout
 -- Full enumeration will populate the cache
 -- None users and groups are expired
Comment 12 Pavel Březina 2017-04-12 10:02:21 UTC 
The same situation may be in sudo rules, anyone who will work on this ticket should investigate.
Comment 13 Pavel Březina 2017-04-12 10:04:08 UTC 
Upstream ticket: https://pagure.io/SSSD/sssd/issue/3369
Comment 14 Lukas Slebodnik 2017-04-12 10:09:35 UTC 
(In reply to Pavel Březina from comment #12)
> The same situation may be in sudo rules, anyone who will work on this ticket
> should investigate.

sudo rules does not have timestamp cache.
Comment 17 Jakub Hrozek 2017-05-02 20:31:05 UTC 
*** Bug 1394295 has been marked as a duplicate of this bug. ***
Comment 20 Jakub Hrozek 2017-05-30 15:40:45 UTC 
Upstream ticket:
https://pagure.io/SSSD/sssd/issue/3369
Comment 24 Jakub Hrozek 2017-06-15 08:06:13 UTC 
*** Bug 1452397 has been marked as a duplicate of this bug. ***
Comment 25 Jakub Hrozek 2017-06-15 09:09:04 UTC 
master:
 * 05e579691b51ac2f81ab0c828ff6fe57bd86a8b6
 * 41708e1e500e7cada3d3e606aa2b8b9869a5c734
 * a71f1a655dcc2ca6dc16bb8eb1c4c9e24cfe2c3e
 * 9883d1e2913ff0c1db479f1ece8148e03155c7f3
 * 8ad57e17779b3ec60246ac58c1691ee15745084c
 * 347be58e1769ba90b49a7e5ec1678ef66987f6cd
 * 01c6bb9b47401f9f14c4cfe5c5f03fce2e63629b
Comment 27 shridhar 2017-06-19 18:18:35 UTC 
verified with 
 ~]# rpm -q sssd
sssd-1.15.2-47.el7.x86_64
[root@shr7-permanent ~]# cat /etc/sssd/sssd.conf 

[sssd]
domains = childb.sssd16.qe
config_file_version = 2
services = nss, pam

[domain/childb.sssd16.qe]
ad_domain = childb.sssd16.qe
krb5_realm = CHILDB.SSSD16.QE
realmd_tags = manages-system joined-with-adcli 
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad
ad_enabled_domains = first.sssd16.qe, childb.sssd16.qe
debug_level = 9

enumerate = true
ldap_enumeration_refresh_timeout = 30
ldap_purge_cache_timeout = 60
entry_cache_timeout = 20




~]# service sssd stop ; rm -rf /var/lib/sss/db/* ; rm -rf /var/log/sssd/* ; date ; service sssd start 
Redirecting to /bin/systemctl stop sssd.service
Mon Jun 19 10:15:38 EDT 2017
Redirecting to /bin/systemctl start sssd.service
[root@shr7-permanent ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb 2> /dev/null |tail -n5
 d16.qe,cn=sysdb

# returned 49 records
# 49 entries
# 0 referrals
[root@shr7-permanent ~]# sleep 60
[root@shr7-permanent ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb |egrep cbu3
asq: Unable to register control with rootdse!
member: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
memberuid: cbu3.qe
dn: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
fullName: cbu3
gecos: cbu3
name: cbu3.qe
originalDN: CN=cbu3,CN=Users,DC=childb,DC=sssd16,DC=qe
userPrincipalName: cbu3.QE
nameAlias: cbu3.qe
distinguishedName: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=
[root@shr7-permanent ~]# egrep cleanup_user /var/log/sssd/sssd_childb.sssd16.qe.log 
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
[root@shr7-permanent ~]# sleep 60
[root@shr7-permanent ~]# egrep cleanup_user /var/log/sssd/sssd_childb.sssd16.qe.log 
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
[root@shr7-permanent ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb |egrep cbu3
asq: Unable to register control with rootdse!
member: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
memberuid: cbu3.qe
memberuid: cbu3.qe
dn: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
fullName: cbu3
gecos: cbu3
name: cbu3.qe
originalDN: CN=cbu3,CN=Users,DC=childb,DC=sssd16,DC=qe
userPrincipalName: cbu3.QE
nameAlias: cbu3.qe
distinguishedName: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=

 ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb 2> /dev/null |tail -n5
 d16.qe,cn=sysdb

# returned 49 records
# 49 entries
# 0 referrals


[root@shr7-permanent ~]# date
Mon Jun 19 10:22:03 EDT 2017
[root@shr7-permanent ~]# egrep cleanup_user /var/log/sssd/sssd_childb.sssd16.qe.log 
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:18:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:18:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:20:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:20:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:21:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:21:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
Comment 31 errata-xmlrpc 2017-08-01 09:04:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2294