oh yeah, and the sssd_kcm process is really hogging the cpu when this occur:

From 'ps':

root      10286 47.5  2.3 199492 41440 ?        Rs   12:01  18:07 /usr/libexec/sssd/sssd_kcm --uid 0 --gid 0 --logger=files

From 'top':
10286 root      20   0  199492  41440   8312 R  73.7   2.3  18:29.80 sssd_kcm 

Yikes, 73%cpu. 

Restart of sssd-kcm (service and socket) does not fix this, but if I do a 'kdestroy -A' kcm cpu usage normalizes. So KCM also seems upset about the state of my credential caches...

Do you use a graphical session (gnome) or a headless server?

Can you enable KCM debugging and add the sssd logs?

It should be enough to drop a snippet with

[kcm]
debug_level = 8

to /etc/sssd.conf.d/ and restart the kcm service.

This is on a headless server. 

I will try to capture a debug output.

So the problem appeared again this morning, attaching kcm debug log. I logged in at 05.33.

Interesting thing I noticed this time is that I actually had two valid credential caches (checking with 'klist -A') - one for the session I just started but one other which had a start time from the middle of the night, which I presume is a renewal made by SSSD. Apart from those I hade 3 other caches with expired tickets, which SSSD apparantly has not renewed.

Created attachment 1572328 [details]
sssd_kcm debug log

It is an interesting observation with the renewal, can you also attach your sssd.conf? But keep in mind that so far sssd only renews tickets which it acquires itself, meaning only tickets acquired through a PAM login, not tickets acquired through kinit or another operation going through libkrb5 directly.

Sure, attaching. 

Regarding renewals I thought that the point of KCM was that SSSD should be able to renew tickets regardless of how they were initiated?

Created attachment 1572366 [details]
sssd.conf

(In reply to adam winberg from comment #7)
> Sure, attaching. 
> 
> Regarding renewals I thought that the point of KCM was that SSSD should be
> able to renew tickets regardless of how they were initiated?

Yes, but this bit has not been implemented yet.

So at this point, is there any benefit of using kcm Vs just disabling it?

(In reply to adam winberg from comment #10)
> So at this point, is there any benefit of using kcm Vs just disabling it?

Not much, unless you care about sharing ccaches between containers by bind-mounting the socket between containers and/or UID namespacing, also typically for the container case.

Ok, interesting. Then I guess I will disable it for now. 

Is there any roadmap for sssd being able to renew tickets 'freely' in RHEL8?

(In reply to adam winberg from comment #12)
> Ok, interesting. Then I guess I will disable it for now. 
> 
> Is there any roadmap for sssd being able to renew tickets 'freely' in RHEL8?

8.1 or 8.2

This upstream ticket is the same issue:

https://pagure.io/SSSD/sssd/issue/4017

As I wrote in the upstream ticket, renewing credentials in the cache won't solve this problem.

In a credential cache that supports collections, only one credential can be active. And for both the kernel persistent keyring and KCM, all processes with the same uid share the same credential cache.

This means that creating an additional credential in the cache, in an attempt to avoid overwriting the credential that is already in the cache, is an error. It's meaningless: if you don't make the new credential the active one, then the process that requested the forwarding will not see the credential that it forwarded. If you *do* make it active, then you have switched the credential from every other process running as that user on the system. Effectively, you've overwritten the previous credential.

KCM needs to implement the logic I described in the upstream bug report: if a credential is being forwarded into KCM, and that credential already exists in the cache, then KCM must make a decision whether to keep the existing credential, or overwrite that credential with the credential that is being forwarded. That decision could be an unconditional "always prefer the incoming credential over the preexisting credential", or KCM could be smarter and bias its selection based on the credentials' respective remaining lifetime. But it *must* make a decision.

And there's another good reason to prefer KCM over the kernel persistent keyring for storing credentials: cifs.upcall is completely ignorant of SELinux, and will cheerfully initialize root's kernel persistent keyring credential cache with an SELinux context that will subsequently deny access to rpc.gssd, which runs in the gssd_t context and cannot access kernel keyrings unless they are properly labeled.

This means that if you configure a Linux host to both access Windows SMB shares via the cifs kernel module and access NFSv4 mounts via Kerberized NFSv4, you have to put gssd_t into permissive mode, because if cifs.upcall initializes (or renews) root's credential cache, it will block access to rpc.gssd. Being forced to put a process that is charge of managing security contexts and credentials into permissive mode to avoid SELinux breakage is unpalatable, so say the least.

In contrast, KCM avoids this issue, because KCM manages its keystore itself, in the filesystem, in a file that has correct SELinux labels. That's a compelling advantage over using the kernel persistent keyring, even if you aren't running containers.

master
        0216bfe - KCM: Fill empty cache, do not initialize a new one
        f5f7f26 - KCM: Allow modifications of ccache's principal
        6b05700 - KCM: Add a forgotten return

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3651

It doesn't look like this has been fully addressed upstream.  And I'm still seeing multiple caches and large sssd_kcm cpu usage on Fedora 31 with sssd-kcm-2.2.3-13.fc31.x86_64.

See followup bug 1780308