This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1900973 - sssd-kcm does not appear to expire Kerberos tickets (RFE: sssd_kcm should have the option to automatically delete the expired tickets)
Summary: sssd-kcm does not appear to expire Kerberos tickets (RFE: sssd_kcm should hav...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: sssd
Version: 9.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Alejandro López
QA Contact: Jakub Vavra
URL:
Whiteboard: sync-to-jira
: 2088901 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-24 06:22 UTC by adam winberg
Modified: 2023-09-18 23:52 UTC (History)
27 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1669607
Environment:
Last Closed: 2023-09-18 23:52:57 UTC
Type: Story
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github SSSD sssd issues 6667 0 None open KCM: provide mechanism to purge expired credentials 2023-04-14 10:37:09 UTC
Red Hat Issue Tracker   RHEL-4971 0 None Migrated None 2023-09-18 23:52:11 UTC
Red Hat Issue Tracker SSSD-6220 0 None None None 2023-06-01 14:15:39 UTC

Description adam winberg 2020-11-24 06:22:45 UTC 
+++ This bug was initially created as a clone of Bug #1669607 +++

Since the upgrade to Fedora 29, my Kerberos tickets no longer expire.  They are kept in the cache indefinitely, it appears.

klist says:

Ticket cache: KCM:1000:512

So I believe sssd-kcm is in fact used.  (I do not otherwise use sssd.)  With Fedora 28, I used the kernel keyring, so I couldn't observe the issue there even if the bug was present.  (This is not an update, but a new installation.)

This affects both FEDORAPROJECT.ORG tickets and REDHAT.COM tickets.  Right now, I don't have any such lingering tickets, but I can run specific diagnostic steps tomorrow if you tell me what to watch out for.

(And I think that keeping session keys around for longer than necessary is a minor security risk.)

Seen with:

sssd-kcm-2.0.0-5.fc29.x86_64
krb5-libs-1.16.1-25.fc29.x86_64

--- Additional comment from Lukas Slebodnik on 2019-01-26 12:15:00 UTC ---

(In reply to Florian Weimer from comment #0)
> Since the upgrade to Fedora 29, my Kerberos tickets no longer expire.  They
> are kept in the cache indefinitely, it appears.
> 
> klist says:
> 
> Ticket cache: KCM:1000:512
> 
> So I believe sssd-kcm is in fact used.  (I do not otherwise use sssd.)  With
> Fedora 28, I used the kernel keyring, so I couldn't observe the issue there
> even if the bug was present.  (This is not an update, but a new
> installation.)
>
sssd.service and sssd-kcm.service are two independent daemons which shares some common internal libraries. 
 
> This affects both FEDORAPROJECT.ORG tickets and REDHAT.COM tickets.  Right
> now, I don't have any such lingering tickets, but I can run specific
> diagnostic steps tomorrow if you tell me what to watch out for.
> 
> (And I think that keeping session keys around for longer than necessary is a
> minor security risk.)
> 

It is a nice idea to have an option for sssd-kcm to destroy tickets at the end of user session.
But on the other hand I like the feature that my ticket is available after reboot. (That was not possible with kernel keyring or with files in /tmp)

> Seen with:
> 
> sssd-kcm-2.0.0-5.fc29.x86_64
> krb5-libs-1.16.1-25.fc29.x86_64

There is a simple way how to use KEYRING ccache if you want. You just need to modify configuration snippet provided by sssd-kcm package. Or you can even uninstall that package.

sh$ rpm -qf /etc/krb5.conf.d/kcm_default_ccache 
sssd-kcm-2.0.0-6.fc30.x86_64

--- Additional comment from Florian Weimer on 2019-01-26 20:12:05 UTC ---

(In reply to Lukas Slebodnik from comment #1)

> > This affects both FEDORAPROJECT.ORG tickets and REDHAT.COM tickets.  Right
> > now, I don't have any such lingering tickets, but I can run specific
> > diagnostic steps tomorrow if you tell me what to watch out for.
> > 
> > (And I think that keeping session keys around for longer than necessary is a
> > minor security risk.)
> > 
> 
> It is a nice idea to have an option for sssd-kcm to destroy tickets at the
> end of user session.

That wouldn't make a difference in my case.

> But on the other hand I like the feature that my ticket is available after
> reboot. (That was not possible with kernel keyring or with files in /tmp)

A ticket that's expired has only value to a potential attacker.  It can only be used to decrypt old data, new data will not be accepted by a peer (because it will observe that its version of the ticket is expired).

> There is a simple way how to use KEYRING ccache if you want. You just need
> to modify configuration snippet provided by sssd-kcm package. Or you can
> even uninstall that package.
> 
> sh$ rpm -qf /etc/krb5.conf.d/kcm_default_ccache 
> sssd-kcm-2.0.0-6.fc30.x86_64

Does this mean you do not consider this a bug?  Thanks.

--- Additional comment from Simo Sorce on 2019-01-27 17:25:55 UTC ---

Florian,
it is generally not a bug.
In krb5, most ccache types do not actively remove expired tickets, until you kdestroy.
The keyring has a nice feature wherby you can make it forget keys automatically, and it also has a limitation on the quantity of data a user can store, so we used that feature. The fact old tickets were removed was just a nice side-effect, but the absence of that behavior is not a bug.
It may a nice feature though, so we could turn this bug into a RFE.

--- Additional comment from Ben Cotton on 2019-08-13 16:57:35 UTC ---

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

--- Additional comment from Ben Cotton on 2019-08-13 19:33:13 UTC ---

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

--- Additional comment from Anthony Messina on 2019-10-27 18:41:09 UTC ---

(In reply to Simo Sorce from comment #3)
> Florian,
> it is generally not a bug.
> In krb5, most ccache types do not actively remove expired tickets, until you
> kdestroy.
> The keyring has a nice feature wherby you can make it forget keys
> automatically, and it also has a limitation on the quantity of data a user
> can store, so we used that feature. The fact old tickets were removed was
> just a nice side-effect, but the absence of that behavior is not a bug.
> It may a nice feature though, so we could turn this bug into a RFE.

The "nice side-effect" also allowed KRB/NFS mounted /home directories (with gssproxy) to work properly since the ccache was always clean after a reboot.  At this point, sssd-kcm integration with gssproxy and nfs-utils is troublesome at best.  To login to a freshly booted workstation (KDE/SDDM), I need to login Ctrl+F2, login to the console (where it tells me I don't have access to my /home directory), then kdestroy -A && kinit, Ctrl+F1, then login via SDDM.  I'm considering (for specific workstations) mounting over /var/lib/sss/secrets with a temporary directory to restore the old "nice side-effect" of starting from scratch on every reboot.

--- Additional comment from Anthony Messina on 2019-10-27 19:10:20 UTC ---

(In reply to Anthony Messina from comment #6)
> (In reply to Simo Sorce from comment #3)
> > Florian,
> > it is generally not a bug.
> > In krb5, most ccache types do not actively remove expired tickets, until you
> > kdestroy.
> > The keyring has a nice feature wherby you can make it forget keys
> > automatically, and it also has a limitation on the quantity of data a user
> > can store, so we used that feature. The fact old tickets were removed was
> > just a nice side-effect, but the absence of that behavior is not a bug.
> > It may a nice feature though, so we could turn this bug into a RFE.
> 
> The "nice side-effect" also allowed KRB/NFS mounted /home directories (with
> gssproxy) to work properly since the ccache was always clean after a reboot.
> At this point, sssd-kcm integration with gssproxy and nfs-utils is
> troublesome at best.  To login to a freshly booted workstation (KDE/SDDM), I
> need to login Ctrl+F2, login to the console (where it tells me I don't have
> access to my /home directory), then kdestroy -A && kinit, Ctrl+F1, then
> login via SDDM.  I'm considering (for specific workstations) mounting over
> /var/lib/sss/secrets with a temporary directory to restore the old "nice
> side-effect" of starting from scratch on every reboot.

The following synthesizes startup/reboot with empty ccaches for everyone/everything

# /etc/tmpfiles.d/sssd-kcm-remove-secrets.conf
# Remove sssd-kcm secrets database on boot
r! /var/lib/sss/secrets/secrets.ldb

With the above, after a reboot:

1. The first time I login via SDDM, sddm-helper fails with: chdir( /home/user1 ) failed for user: "user1", verify directory exist and has sufficient permissions
2. I am returned to the login screen
3. The second time I login via SDDM, it works properly

It seems the first go-round initializes the KCM: ccache with the nfs/sever@REALM ticket, which is then "ready" for the second login attempt.

It's likely none of this is helpful, but wanted to log end-user usability issues with the default Fedora setup with sssd-kcm, gssproxy, nfs-utils.

--- Additional comment from Orion Poplawski on 2020-06-01 21:40:43 UTC ---

I'm not sure the non-removal of expired ticket is *directly* related to the NFS issues, or if it more of an issue how KCM handles cache collections in general.  Is this just another manifestation of https://github.com/SSSD/sssd/issues/4988 ?

--- Additional comment from Fedora Admin user for bugzilla script actions on 2020-06-18 14:59:25 UTC ---

This package has changed maintainer in the Fedora.
Reassigning to the new maintainer of this component.

--- Additional comment from Pavel Březina on 2020-07-08 13:03:17 UTC ---

This seems to be corresponding upstream ticket:
https://github.com/SSSD/sssd/issues/3593

--- Additional comment from Florian Weimer on 2020-07-08 13:08:18 UTC ---

(In reply to Pavel Březina from comment #10)
> This seems to be corresponding upstream ticket:
> https://github.com/SSSD/sssd/issues/3593

It's perhaps related, but slightly different. Expired tickets are only useful to attackers, so there's absolutely no reason to keep them around. Purging tickets on session end might impact some workflows negatively.

--- Additional comment from Robbie Harwood on 2020-07-08 17:35:38 UTC ---

> Expired tickets are only useful to attackers, so there's absolutely no reason to keep them around.

That's... more reductionist than I think is accurate.

krb5 does change behavior based on the presence of tickets, expired or no.  For instance, consider a collection with credentials for REDHAT.COM and IPA.REDHAT.COM (in that order).  For hostname.redhat.com, the credential for REDHAT.COM will typically be preferred, even if it's expired.  Pruning the credential for REDHAT.COM will cause the one for IPA.REDHAT.COM to be used instead.  In that case, pruning is desirable (and for this reason, Simo has been in favor of it I believe).

However, consider instead the same setup with REDHAT.COM and FEDORAPROJECT.ORG.  Pruning expired REDHAT.COM will cause FEDORAPROJECT.ORG's credential to be attempted.  While this will obviously not work (no cross realm there), the errors will be confusing to a user who doesn't realize the REDHAT.COM credential has been expired (and there's nothing krb5 can do about it because the ccache expired it out from under us).  And it's potentially even more confusing when cross-realm relationships come into play.

Upstream's position is that the second case is more typical, so in-tree ccache backends (FILE, DIR, MEMORY, etc.) do not prune.  KEYRING prunes.
Comment 1 adam winberg 2020-11-24 06:35:36 UTC 
> The keyring has a nice feature wherby you can make it forget keys automatically, and it also has a limitation on the quantity of data a user can store, so we used that feature. The fact old tickets were removed was just a nice side-effect, but the absence of that behavior is not a bug.

Is it not a bit odd to incorporate the quota feature from keyring but not the automatic purging? Does this not just inevitably lead to quota being full when old credentials are never removed?

At least that's what I'm running into, I think. Suddenly a user can no longer get any new kerberos credentials and therefore cant access the nfs homedir. A 'klist' shows a long list of expired credentials, purging with 'kdestroy -A' and the user gets new creds on login again. 

To remedy this we have built a cron script which refreshes and purges users kerberos credentials. Feels crazy though that we should have to do that. 
However, users with keytabs which get credentials via gssproxy cant make use of that workaround. Credentials from gssproxy are obscured with klist:

[user.u@lxserv2114 ~]$ klist
Ticket cache: KCM:17098:66803
Default principal: user.u@AD

Valid starting       Expires              Service principal
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00  01/01/1970 00:00:00  Encrypted/Credentials/v1@X-GSSPROXY:


So there's no way to purge expired credentials for this user other than 'kdestroy -A' which of course also removes active credentials. 
The klist output above is from a user which suddenly no longer can log in because no new credentials are given. After a 'kdestroy -A' it works again. 

There are no logs in sssd_kcm.log when this happens. I will try to reproduce with a higher debug_level though. If the problem is 'max_ccache_size' I could of course increase this, but that would just delay the problem since the credentials are never removed, just added.
Comment 2 adam winberg 2020-11-24 07:05:56 UTC 
So i recreated the error which is not hard to do:

1. Log in to remote server. A new credential is issued
2. Log out. 
3. Log in to remote server. A new credential is issued
4. Repeat

When the number of credential caches for my user hits ~64 I no longer get any new credentials on login. sssd_kcm.log (with debug_level=8) shows:

(2020-11-24  6:54:17): [kcm] [local_decrypt] (0x2000): Decrypting with masterkey
(2020-11-24  6:54:17): [kcm] [sss_sec_update] (0x0400): Adding a secret to [persistent/61526/default]
(2020-11-24  6:54:17): [kcm] [local_db_check_peruid_number_of_secrets] (0x0040): Cannot store any more secrets for this client (basedn cn=61526,cn=persistent,cn=kcm) as the maximum allowed limit (66) has been reached
(2020-11-24  6:54:17): [kcm] [sss_sec_update] (0x0040): local_db_check_number_of_secrets failed [1432158287]: The maximum number of stored secrets has been reached
(2020-11-24  6:54:17): [kcm] [sec_update_b64] (0x0040): Cannot write the secret [1432158287]: The maximum number of stored secrets has been reached
(2020-11-24  6:54:17): [kcm] [kcm_ccdb_set_default_done] (0x0040): Failed to set the default ccache [1432158287]: The maximum number of stored secrets has been reached
(2020-11-24  6:54:17): [kcm] [kcm_op_set_default_done] (0x0040): Cannot set default ccache 1432158287: The maximum number of stored secrets has been reached
(2020-11-24  6:54:17): [kcm] [kcm_cmd_done] (0x0040): op receive function failed [1432158287]: The maximum number of stored secrets has been reached
(2020-11-24  6:54:17): [kcm] [kcm_cmd_request_done] (0x0040): KCM operation failed [1432158287]: The maximum number of stored secrets has been reached
(2020-11-24  6:54:17): [kcm] [kcm_reply_error] (0x0040): KCM operation returs failure [1432158287]: The maximum number of stored secrets has been reached

Increasing the 'max_uid_ccaches' parameter allows me to once again get new credentials. But since the credentials are never removed this is not a solution. 

Our workaround right now, as I said, is a cron script purging active users credentials. But that does not work for the keytab users getting credentials via gssproxy.
Comment 3 Luc de Louw 2021-01-13 21:16:42 UTC 
To be able to reproduce the log entries, add the following to /etc/sssd/sssd.conf:

[kcm]
debug_level = 9
Comment 11 jstephen 2022-06-07 18:39:48 UTC 
*** Bug 2088901 has been marked as a duplicate of this bug. ***
Comment 22 Alejandro López 2023-08-10 09:26:38 UTC 
The fix we decided to implement is to remove the oldest expired ticket whenever a new ticker is to be stored and there is no more free space.
Comment 25 RHEL Program Management 2023-09-18 23:47:54 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 26 RHEL Program Management 2023-09-18 23:52:57 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
Note You need to log in before you can comment on or make changes to this bug.