RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1430415 - ldap_purge_cache_timeout in RHEL7.3 invalidate most of the entries once the cleanup task kicks in
Summary: ldap_purge_cache_timeout in RHEL7.3 invalidate most of the entries once the c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sssd
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: SSSD Maintainers
QA Contact: shridhar
URL:
Whiteboard:
: 1394295 1452397 (view as bug list)
Depends On:
Blocks: 1420851 1452916 1471808
TreeView+ depends on / blocked
 
Reported: 2017-03-08 14:51 UTC by Ming Davies
Modified: 2020-12-14 08:18 UTC (History)
14 users (show)

Fixed In Version: sssd-1.15.2-47.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1471808 (view as bug list)
Environment:
Last Closed: 2017-08-01 09:04:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
latest sssd debug (6.66 MB, application/x-bzip)
2017-04-11 13:43 UTC, Ming Davies 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github SSSD sssd issues 4399 0 None closed ldap_purge_cache_timeout in RHEL7.3 invalidate most of the entries once the cleanup task kicks in 2020-12-18 09:51:45 UTC
Red Hat Product Errata RHEA-2017:2294 0 normal SHIPPED_LIVE sssd bug fix and enhancement update 2017-08-01 12:39:55 UTC

Description Ming Davies 2017-03-08 14:51:11 UTC 
Description of problem:
Customer said:"Our intention is to have the entire ldap catalog in the local cache and for that purpose we have set "enumerate = true" but but every 3 hours when ldap_purge_cache_timeout is done all accounts, groups and sudo rules is removed from the local cache. After 3 hours again all entries is back in the local cache". The customer is fully aware of the performance impact when using enumerate.

The customer claims that the issue is perceived on servers running 7.3 and sssd 1.14.0 release 43.el7_3.11, but seemed to work fine in rhel 6 and sssd 1.13.3 rel 22.


According to "man sssd-ldap", if enumeration is enabled, the cleanup task, i.e. ldap_purge_cache_timeout is required in order to detect entries removed from the server and can't be disabled!



Version-Release number of selected component (if applicable):
sssd 1.14.0 release 43.el7_3.11

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 9 Ming Davies 2017-04-11 13:43:55 UTC 
Created attachment 1270783 [details]
latest sssd debug
Comment 11 Pavel Březina 2017-04-12 10:01:36 UTC 
Hi, thank you for the logs. This is indeed an issue due to introduction of the timestamp cache to speed up performance.

1. Full enumeration fills cache with users and groups
 -- for next three hours enumeration will be usen entryUSN to fetch only new entries
2. Purge cache timeout kicks in
 -- Full enumeration is done, but it only updates timestamp cache
 -- We search users in data cache with expiriration time filter
 -- We end up deleting all users
3. Again only smart enumeration with entryUSN is used, which won't get any result
4. Purge cache timeout
 -- Full enumeration will populate the cache
 -- None users and groups are expired
Comment 12 Pavel Březina 2017-04-12 10:02:21 UTC 
The same situation may be in sudo rules, anyone who will work on this ticket should investigate.
Comment 13 Pavel Březina 2017-04-12 10:04:08 UTC 
Upstream ticket: https://pagure.io/SSSD/sssd/issue/3369
Comment 14 Lukas Slebodnik 2017-04-12 10:09:35 UTC 
(In reply to Pavel Březina from comment #12)
> The same situation may be in sudo rules, anyone who will work on this ticket
> should investigate.

sudo rules does not have timestamp cache.
Comment 17 Jakub Hrozek 2017-05-02 20:31:05 UTC 
*** Bug 1394295 has been marked as a duplicate of this bug. ***
Comment 20 Jakub Hrozek 2017-05-30 15:40:45 UTC 
Upstream ticket:
https://pagure.io/SSSD/sssd/issue/3369
Comment 24 Jakub Hrozek 2017-06-15 08:06:13 UTC 
*** Bug 1452397 has been marked as a duplicate of this bug. ***
Comment 25 Jakub Hrozek 2017-06-15 09:09:04 UTC 
master:
 * 05e579691b51ac2f81ab0c828ff6fe57bd86a8b6
 * 41708e1e500e7cada3d3e606aa2b8b9869a5c734
 * a71f1a655dcc2ca6dc16bb8eb1c4c9e24cfe2c3e
 * 9883d1e2913ff0c1db479f1ece8148e03155c7f3
 * 8ad57e17779b3ec60246ac58c1691ee15745084c
 * 347be58e1769ba90b49a7e5ec1678ef66987f6cd
 * 01c6bb9b47401f9f14c4cfe5c5f03fce2e63629b
Comment 27 shridhar 2017-06-19 18:18:35 UTC 
verified with 
 ~]# rpm -q sssd
sssd-1.15.2-47.el7.x86_64
[root@shr7-permanent ~]# cat /etc/sssd/sssd.conf 

[sssd]
domains = childb.sssd16.qe
config_file_version = 2
services = nss, pam

[domain/childb.sssd16.qe]
ad_domain = childb.sssd16.qe
krb5_realm = CHILDB.SSSD16.QE
realmd_tags = manages-system joined-with-adcli 
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad
ad_enabled_domains = first.sssd16.qe, childb.sssd16.qe
debug_level = 9

enumerate = true
ldap_enumeration_refresh_timeout = 30
ldap_purge_cache_timeout = 60
entry_cache_timeout = 20




~]# service sssd stop ; rm -rf /var/lib/sss/db/* ; rm -rf /var/log/sssd/* ; date ; service sssd start 
Redirecting to /bin/systemctl stop sssd.service
Mon Jun 19 10:15:38 EDT 2017
Redirecting to /bin/systemctl start sssd.service
[root@shr7-permanent ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb 2> /dev/null |tail -n5
 d16.qe,cn=sysdb

# returned 49 records
# 49 entries
# 0 referrals
[root@shr7-permanent ~]# sleep 60
[root@shr7-permanent ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb |egrep cbu3
asq: Unable to register control with rootdse!
member: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
memberuid: cbu3.qe
dn: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
fullName: cbu3
gecos: cbu3
name: cbu3.qe
originalDN: CN=cbu3,CN=Users,DC=childb,DC=sssd16,DC=qe
userPrincipalName: cbu3.QE
nameAlias: cbu3.qe
distinguishedName: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=
[root@shr7-permanent ~]# egrep cleanup_user /var/log/sssd/sssd_childb.sssd16.qe.log 
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
[root@shr7-permanent ~]# sleep 60
[root@shr7-permanent ~]# egrep cleanup_user /var/log/sssd/sssd_childb.sssd16.qe.log 
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
[root@shr7-permanent ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb |egrep cbu3
asq: Unable to register control with rootdse!
member: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
memberuid: cbu3.qe
memberuid: cbu3.qe
dn: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=sysdb
fullName: cbu3
gecos: cbu3
name: cbu3.qe
originalDN: CN=cbu3,CN=Users,DC=childb,DC=sssd16,DC=qe
userPrincipalName: cbu3.QE
nameAlias: cbu3.qe
distinguishedName: name=cbu3.qe,cn=users,cn=childb.sssd16.qe,cn=

 ~]# ldbsearch -H /var/lib/sss/db/cache_childb.sssd16.qe.ldb 2> /dev/null |tail -n5
 d16.qe,cn=sysdb

# returned 49 records
# 49 entries
# 0 referrals


[root@shr7-permanent ~]# date
Mon Jun 19 10:22:03 EDT 2017
[root@shr7-permanent ~]# egrep cleanup_user /var/log/sssd/sssd_childb.sssd16.qe.log 
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:15:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:17:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:18:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:18:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:20:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:20:08 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
(Mon Jun 19 10:21:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x4000): Cache expiration is set to 0 days
(Mon Jun 19 10:21:38 2017) [sssd[be[childb.sssd16.qe]]] [cleanup_users] (0x0200): Found 0 expired user entries!
Comment 31 errata-xmlrpc 2017-08-01 09:04:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2294
Note You need to log in before you can comment on or make changes to this bug.