Note that BZ#1712875 (Old kerberos credentials active instead of valid new ones) did not resolve this issue.

With BZ#1712875 in place, when ssh delegates credentials to a remote host that is using KCM, the newly-delegated credentials are made active, but old credentials (if present) remain in the cache. While that addresses the issue of the user not having access to resources because sssd did not switch to the newly-delegated credential, it does not address the other problems that result when the credential cache contains multiple identical principals.

For example, depending on the ordering of credentials in the cache, invoking "kswitch -p PRINCPAL" can select old, expired credentials in the cache. This is because kswitch does not expect the credential cache to contain multiple identical principals, and provides no mechanism to refine the selection of the principal that "kswitch -p PRINCIPAL" selects if multiple principals are present. This is understandable, considering that of the credential cache types that support collections, neither kernel persistent keyrings nor the DIR: credential cache support duplicate credentials.

Again, as per the description above: KCM must not create multiple principals in the cache. If an incoming connection is attempting to delegate a credential for a principal that already has a credential in the cache, KCM *must not* create a duplicate credential. It must either discard the incoming credential in favor of the one already in the cache, or overwrite the credential already in the cache with the incoming credential.

As per above, it might make sense for sssd to provide a knob (e.g., delegation_acceptance_criteria) that effects how KCM decides which action to take, instead of always overwriting the existing credential with the incoming credential.

(Per our support case, I am adding this requested information.)

Description of problem:

See the Description.

Version-Release number of selected component (if applicable):

sssd-2.2.0-19.el8.x86_64

How reproducible:

Login multiple times to the same host via ssh via gssapi-with-mic or gssapi-keyex authentication with credential forwarding enabled.

Steps to Reproduce:

1. Ensure that sssd-kcm is installed on the target host.

2. Login to the target host via ssh via gssapi-with-mic or gssapi-keyex authentication with credential forwarding enabled and run "kinit -A".  Results will be similar to the following:

$ klist -A
Ticket cache: KCM:116201957:39355
Default principal: username

Valid starting       Expires              Service principal
2020-01-21 15:43:47  2020-01-22 11:59:51  krbtgt/EXAMPLE.ORG
        renew until 2020-01-27 23:38:52

3. Log out, login again (via ssh via gssapi-with-mic or gssapi-keyex), and run "klist -A" again:

$ klist -A
Ticket cache: KCM:116201957:62082
Default principal: username

Valid starting       Expires              Service principal
2020-01-21 15:44:16  2020-01-22 11:59:51  krbtgt/EXAMPLE.ORG
        renew until 2020-01-27 23:38:52

Ticket cache: KCM:116201957:39355
Default principal: username

Valid starting       Expires              Service principal
2020-01-21 15:43:47  2020-01-22 11:59:51  krbtgt/EXAMPLE.ORG
        renew until 2020-01-27 23:38:52

Actual results:

You will see that KCM has duplicated the credential in the cache.

Expected results:

KCM should overwrite the old credential with the incoming credential (which is how the kernel persistent keyring behaves), discard the incoming credential in favor of the one already in the cache, or provide a configuration option to help influence how KCM selects the credential to keep versus the one to discard.

Additional info:

Having multiple credentials in the cache breaks "kswitch -p". No other Kerberos credential cache type supports multiple credentials in the catch for the exact same principal. sssd shouldn't, either.

This is a problem for us as well.

I strongly believe this is related to this as well: https://bugzilla.redhat.com/show_bug.cgi?id=1645624

If I clear the caches, goa-identity-service does not constantly poll sssd_kcm like crazy trying to renew the keys in the cache.

(In reply to James Ralston from comment #4)
> 
> Actual results:
> 
> You will see that KCM has duplicated the credential in the cache.
> 
> Expected results:
> 
> KCM should overwrite the old credential with the incoming credential (which
> is how the kernel persistent keyring behaves), discard the incoming
> credential in favor of the one already in the cache, or provide a
> configuration option to help influence how KCM selects the credential to
> keep versus the one to discard.
> 
> Additional info:
> 
> Having multiple credentials in the cache breaks "kswitch -p". No other
> Kerberos credential cache type supports multiple credentials in the catch
> for the exact same principal. sssd shouldn't, either.


It looks to me that the persistent Keyring ccache exhibits the same behavior, it does *not* remove the previously delegated credential. 


[vagrant.vm ~]$ ssh -K testuser1.vm kdestroy -A
[vagrant.vm ~]$ ssh -K testuser1.vm klist -A
Ticket cache: KEYRING:persistent:340600004:krb_ccache_ehsVzI9
Default principal: testuser1

Valid starting       Expires              Service principal
09/11/2020 15:30:21  09/12/2020 15:16:53  krbtgt/IPA.VM
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ ssh -K testuser1.vm klist -A
Ticket cache: KEYRING:persistent:340600004:krb_ccache_F1XP75X
Default principal: testuser1

Valid starting       Expires              Service principal
09/11/2020 15:30:23  09/12/2020 15:16:53  krbtgt/IPA.VM

Ticket cache: KEYRING:persistent:340600004:krb_ccache_ehsVzI9
Default principal: testuser1

Valid starting       Expires              Service principal
09/11/2020 15:30:21  09/12/2020 15:16:53  krbtgt/IPA.VM
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ 
[vagrant.vm ~]$ ssh -K testuser1.vm klist -A
Ticket cache: KEYRING:persistent:340600004:krb_ccache_js83pc7
Default principal: testuser1

Valid starting       Expires              Service principal
09/11/2020 15:30:25  09/12/2020 15:16:53  krbtgt/IPA.VM

Ticket cache: KEYRING:persistent:340600004:krb_ccache_F1XP75X
Default principal: testuser1

Valid starting       Expires              Service principal
09/11/2020 15:30:23  09/12/2020 15:16:53  krbtgt/IPA.VM

Ticket cache: KEYRING:persistent:340600004:krb_ccache_ehsVzI9
Default principal: testuser1

Valid starting       Expires              Service principal
09/11/2020 15:30:21  09/12/2020 15:16:53  krbtgt/IPA.VM


The keyring does remove old tickets after some time, could that be the reason you are noticing a difference in behavior with KCM and the kernel keyring ccache?

From re-testing, I believe you are correct: I also see that repeated logins within an arbitrary window of time will generate multiple credentials in the kernel persistent keyring. I suspect I did not realize that the kernel persistent keyring has this behavior because—as you mentioned—the kernel persistent keyring purges old credentials after some period of time.

However, this doesn't change the underlying issue: KCM does not purges old credentials, and when receiving a remote credential via ssh via gssapi-with-mic or gssapi-keyex authentication, KCM always prefers an existing credential in the cache, which is typically older (or outright expired).

I still stand by my original request: KCM should not create a duplicate credentials in the cache. Rather, it should either keep the existing credential in the cache (discarding the incoming credential), or overwrite the existing credential with the incoming credential. There should be an sssd option to control this behavior (e.g., delegation_acceptance_criteria).

Additionally, based on Simone's report in BZ#1645624, even if the above logic is implemented, it sounds like KCM really needs a background credential remover, similar to what the kernel persistent keyring has, to remove expired credentials.

(In reply to James Ralston from comment #10)

Hi James, thanks a lot for your inputs.

> From re-testing, I believe you are correct: I also see that repeated logins
> within an arbitrary window of time will generate multiple credentials in the
> kernel persistent keyring. I suspect I did not realize that the kernel
> persistent keyring has this behavior because—as you mentioned—the kernel
> persistent keyring purges old credentials after some period of time.
> 
> However, this doesn't change the underlying issue: KCM does not purges old
> credentials, and when receiving a remote credential via ssh via
> gssapi-with-mic or gssapi-keyex authentication, KCM always prefers an
> existing credential in the cache, which is typically older (or outright
> expired).

Could you please double-check and confirm that this issue still occurs on sssd-kcm-2.2.0-19.el8 or higher, specifically the issue where KCM does not switch to the newly delegated credentials, but expired credentials are chosen instead.

The reason I ask is because in my testing I am seeing that KCM is always switching to the updated credentials when newly received remote credentials are delegated over SSH with GSSAPI, and the behavior you described is exactly the behavior which should have been fixed. Excerpt from the upstream PR comment: "...Afterwards, klist or similar would see that the default ccache is empty and just pick the first non-empty one as a fallback, which would often be one of the previous expired ones."

(Note, I am testing in my Fedora kerberized NFS environment with sssd-2.3.1-4.fc33.x86_64, this includes the fixes from https://bugzilla.redhat.com/show_bug.cgi?id=1712875 and no other KCM changes should make a difference in these codepaths)

If the preferring expired credentials problem persists then please provide 'debug_level=9' debug logs from sssd-kcm on the targethost. I would like to look for logs similar to:

   [kcm_op_set_default_create_step_done] (0x0400): The ccache was created, switching to it
   [kcm_op_initialize_got_default] (0x0400): The default ccached was not set, switching to the initialized

> 
> I still stand by my original request: KCM should not create a duplicate
> credentials in the cache. Rather, it should either keep the existing
> credential in the cache (discarding the incoming credential), or overwrite
> the existing credential with the incoming credential. There should be an
> sssd option to control this behavior (e.g., delegation_acceptance_criteria).

For now, please ignore the duplicate credentials aspect of this problem, I would like to focus on ensuring KCM is switching to newly delegated credentials and not selecting expired ccache credentials.
> 
> Additionally, based on Simone's report in BZ#1645624, even if the above
> logic is implemented, it sounds like KCM really needs a background
> credential remover, similar to what the kernel persistent keyring has, to
> remove expired credentials.

Hi Justin,

From preliminary testing, I believe you are correct: as of at least sssd 2.2.3-20.el8, KCM seems to unconditionally switch to the incoming credential. So, this is definitely an improvement.

For now, I'll clear the NEEDINFO flag. We've configured our beta RHEL8 hosts to use KCM instead of the kernel persistent keyring. I'll check back in a week or so and report what problems (if any) we've noticed.

As for credential cleanups…

With the kernel persistent keyring, we set "GSSAPICleanupCredentials no" in /etc/ssh/sshd_config, because we didn't want different login sessions potentially wiping out each other's credentials. But if KCM always caches and switches to the incoming credential, then perhaps a better thing to do here is to set "GSSAPICleanupCredentials yes", so that sshd removes the credential cache when the ssh session closes.

But: that won't clean credentials that are created when a user logs in via any other mechanism than ssh with gssapi-with-mic or gssapi-keyex authentication. That's an issue for us, as a fair number of our users login to various hosts via ssh with keyboard-interactive authentication.

Setting "ccache_storage = memory" in the [kcm] section of sssd.conf could provide another mechanism to clean old credentials, but only because *all* credentials are lost when the host reboots (or sssd otherwise shuts down). But there's no guarantee that a host will reboot regularly, and flushing *all* cached credentials upon reboot is often undesirable.

So I think there is still a need for KCM to periodically clean expired credentials from the cache, both for credentials it creates itself, and for credentials created by other applications.

Thank you James for the quick response and testing confirmation. Please let me know how the results go in your testing. If you are no longer seeing exising expired credentials being switched to by KCM then I would like to close out this BZ, and to avoid confusion keep any residual issues separated out into different tickets.

We have an existing upstream ticket[1] which could help to address the issue with cleaning non-GSSAPI managed credentials. In addition, the issues you mention could be mitigated by adding KCM session support[2] to enforce isolation of different login sessions per user. I could see this as an improvement to the GSSAPI ssh-managed ccache use cases with KCM. I can also create an upstream ticket for expiring tickets in KCM if you would like.

[1] https://github.com/SSSD/sssd/issues/3593

[2] https://github.com/SSSD/sssd/issues/5334

Hi Justin,

We haven't noticed any instances of KCM failing to set the active credential cache to the incoming credentials, so I think this ticket can be closed. Thanks for your work on this!

I agree that cleaning unused and/or expired credentials from the cache should be a separate issue.

Also, to clarify comment 10 with respect to the behavior of the kernel persistent keyring:

• If multiple incoming ssh connections for the same user are opened on a RHEL7 host, all sessions share the same credential in the cache.

• If multiple incoming ssh connections for the same user are opened on a RHEL8 host, each session gets a new credential in the cache.

So, between RHEL7 and RHEL8, the behavior of the kernel persistent keyring cache changed. I don't know what ultimately caused that change, though: it might have been changes in the kernel persistent keyring itself, changes to kernel tunables (which we can potentially revert), changes to the behavior of OpenSSH, or some combination of these.

Regardless, on RHEL7, we must continue to set "GSSAPICleanupCredentials no" in /etc/ssh/sshd_config if the kernel persistent keyring is in use, because otherwise different login sessions will wipe out each other's credentials.  On RHEL8, it is more appropriate to use "GSSAPICleanupCredentials yes" (the default) in /etc/ssh/sshd_config, for both the kernel persistent keyring and for KCM, because both credential storage types create duplicate credentials in the cache for new sessions.  (So when a session terminates and sshd cleans the credentials, that won't break other sessions.)

This was confirmed as fixed in https://github.com/SSSD/sssd/pull/876 and included in sssd-2.2.0-19.el8 and later.