Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2143925

Summary: kinit switches KCM away from the newly issued ticket
Product: Red Hat Enterprise Linux 8 Reporter: Ian Collier <imc>
Component: sssdAssignee: Alejandro López <allopez>
Status: CLOSED ERRATA QA Contact: Jakub Vavra <jvavra>
Severity: low Docs Contact:
Priority: unspecified    
Version: 8.6CC: aboscatt, atikhono, ddas, jabsher, jstephen, pbrezina, rakkumar, ralston, sgadekar
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: sync-to-jira
Fixed In Version: sssd-2.9.0-2.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-14 15:49:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Debug log from sssd_kcm none

Description Ian Collier 2022-11-18 11:59:59 UTC
Description of problem:
It may be relevant that I have two ssh sessions active to the same headless server.  In one of them if my ticket has expired and I type 'kinit' to reissue it, the primary cache in the KCM collection changes to a different (expired) ticket, so KRB5 operations still fail.

Version-Release number of selected component (if applicable):
krb5-workstation-1.18.2-14.el8.x86_64
sssd-kcm-2.6.2-4.el8_6.1.x86_64

How reproducible:
It happens reasonably often when my tickets expire.

Transcript (the current time is about 11:30 on Nov 18):

$ klist
Ticket cache: KCM:10282:19556
Default principal: imc.OX.AC.UK

17/11/22 12:22:15  17/11/22 18:04:10  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK
        renew until 24/11/22 10:04:10

[The ticket has expired.  Let's issue a new one.]

$ kinit
Password for imc.OX.AC.UK: 
$ klist
Ticket cache: KCM:10282:46955
Default principal: imc.OX.AC.UK

Valid starting     Expires            Service principal
17/11/22 11:38:11  18/11/22 10:42:42  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK

[So, despite having issued a new ticket with 'kinit', the current ticket
is still expired!  The identity of the ticket cache has changed.]

$ kswitch -c KCM:10282:19556
$ klist
Ticket cache: KCM:10282:19556
Default principal: imc.OX.AC.UK

Valid starting     Expires            Service principal
18/11/22 11:33:43  19/11/22 11:33:41  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK

[Oh look, there is my newly issued ticket after all.]

I would contend that the 'kswitch' there shouldn't be needed in order to
have a valid ticket immediately after a successful 'kinit'.

Comment 1 jstephen 2022-11-21 14:36:35 UTC
*** Bug 2140105 has been marked as a duplicate of this bug. ***

Comment 2 jstephen 2022-11-21 14:36:53 UTC
*** Bug 2128507 has been marked as a duplicate of this bug. ***

Comment 3 jstephen 2022-11-21 14:36:57 UTC
*** Bug 2144401 has been marked as a duplicate of this bug. ***

Comment 4 jstephen 2022-11-30 15:04:45 UTC
Hi, thank you for reporting this issue.

Could you please provide the following information when this issue occurs again.

1) KCM debug logs covering the time this issue occurs. First add the 'debug_level' option to sssd.conf, then restart SSSD for the changes to take effect. The KCM logs may get quite large quickly so please be aware to avoid running out of disk space.

   [kcm]
   debug_level = 9

2) 'klist -a' and 'date' command output before AND after the primary ccache changes incorrectly.

Comment 5 James Ralston 2022-12-02 05:20:22 UTC
(In reply to jstephen from comment #4)
> Could you please provide the following information when this issue occurs
> again.
> 
> 1) KCM debug logs …
> 2) 'klist -a' and 'date' command output before AND after …

I have gathered these data, but they contain potentially sensitive information (domains, usernames, uids). Is there a way I can provide these data privately?

(I could also redact the sensitive information, but I am uncertain whether doing so would hinder its usefulness in debugging.)

Thanks!

Comment 6 jstephen 2022-12-02 14:49:05 UTC
(In reply to James Ralston from comment #5)
> (In reply to jstephen from comment #4)
> > Could you please provide the following information when this issue occurs
> > again.
> > 
> > 1) KCM debug logs …
> > 2) 'klist -a' and 'date' command output before AND after …
> 
> I have gathered these data, but they contain potentially sensitive
> information (domains, usernames, uids). Is there a way I can provide these
> data privately?
> 
> (I could also redact the sensitive information, but I am uncertain whether
> doing so would hinder its usefulness in debugging.)
> 
> Thanks!

If you are okay with it, you can send directly to my email at jstephen, thank you!

Comment 7 James Ralston 2022-12-04 22:11:19 UTC
(In reply to jstephen from comment #6)
> If you are okay with it, you can send directly to my email at
> jstephen, thank you!

Sent on 2022-12-02 at 14:15:27-0500 with a Subject of “logs for BZ#2143925”.

Comment 8 Ian Collier 2022-12-04 22:26:55 UTC
(In reply to jstephen from comment #4)

> 1) KCM debug logs covering the time this issue occurs. First add the
> 'debug_level' option to sssd.conf, then restart SSSD for the changes to take
> effect.

The entire content of sssd_kcm.log is the following one line:

(2022-12-04 22:14:06): [kcm] [server_setup] (0x1f7c0): Starting with debug level = 0x0070

> 2) 'klist -a' and 'date' command output before AND after the primary ccache
> changes incorrectly.

$ date;klist -a; kinit;klist -a; date
Sun  4 Dec 22:16:58 GMT 2022
Ticket cache: KCM:10282:19556
Default principal: imc.OX.AC.UK

Valid starting     Expires            Service principal
01/12/22 18:51:12  02/12/22 16:05:08  HTTP/idmsrv1.ipa.cs.ox.ac.uk.OX.AC.UK
        Addresses: (none)
01/12/22 16:22:01  02/12/22 16:05:08  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK
        Addresses: (none)
Password for imc.OX.AC.UK: 
Ticket cache: KCM:10282:16032
Default principal: imc.OX.AC.UK

Valid starting     Expires            Service principal
29/11/22 10:10:35  30/11/22 09:10:59  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK
        Addresses: (none)
29/11/22 10:10:44  30/11/22 09:10:59  HTTP/idmsrv1.ipa.cs.ox.ac.uk.OX.AC.UK
        Addresses: (none)
Sun  4 Dec 22:17:02 GMT 2022


I don't believe sssd is involved at all when this happens because
I'm using 'kinit' from krb5-workstation.

Comment 9 jstephen 2022-12-05 20:53:27 UTC
(In reply to Ian Collier from comment #8)
> (In reply to jstephen from comment #4)
> 
> > 1) KCM debug logs covering the time this issue occurs. First add the
> > 'debug_level' option to sssd.conf, then restart SSSD for the changes to take
> > effect.
> 
> The entire content of sssd_kcm.log is the following one line:
> 
> (2022-12-04 22:14:06): [kcm] [server_setup] (0x1f7c0): Starting with debug
> level = 0x0070

> 
> I don't believe sssd is involved at all when this happens because
> I'm using 'kinit' from krb5-workstation.

You should see a number of log messages written to the sssd_kcm.log file when debug_level is set to a high value. You may need to check the 'sssd-kcm' systemd service, if you are not using sssd outside of KCM.

Comment 15 Ian Collier 2022-12-15 15:11:24 UTC
Created attachment 1932865 [details]
Debug log from sssd_kcm

Sorry about that.  The log message showed sssd_kcm starting up
after I edited the config file with a debug level of 0x0070, which
I now know means it didn't pick up the debug level I'd added to the
config file (no idea why not).  I've now managed to persuade it
to give debug messages, and after a week of waiting for the bug to
reappear, we now have a log.

The sssd_kcm log covers typing 'klist' (when the daemon auto-starts)
and then a few seconds later this sequence:

$ date;klist -a;kinit;klist -a;date
Thu 15 Dec 14:30:34 GMT 2022
Ticket cache: KCM:10282:92862
Default principal: imc.OX.AC.UK

Valid starting     Expires            Service principal
12/12/22 11:31:14  13/12/22 10:57:05  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK
        Addresses: (none)
12/12/22 11:31:36  13/12/22 10:57:05  HTTP/idmsrv1.ipa.cs.ox.ac.uk.OX.AC.UK
        Addresses: (none)
Password for imc.OX.AC.UK: 
Ticket cache: KCM:10282:72015
Default principal: imc.OX.AC.UK

Valid starting     Expires            Service principal
15/12/22 12:31:54  16/12/22 12:31:54  krbtgt/IPA.CS.OX.AC.UK.OX.AC.UK
        Addresses: (none)

Comment 16 Ian Collier 2022-12-15 15:18:43 UTC
Now it turns out that the ticket to which it switched me this time is
still valid.  This ticket would have been generated (at 12:31) by me
doing something in the current session that required 'sudo' and thus
typing my password.

So in fact the bug happened twice today: firstly when typing 'sudo' it
populated the (at that time) default cache of 10282:72015 and then
immediately switched to the (expired) 10282:92862; then when typing
'kinit' it added the ticket to 10282:92862 and switched back to
10282:72015 which still has the ticket generated earlier.

We would have logs for that earlier 'sudo' as well if required, but in
either case it doesn't seem that the reason for the switch was recorded
in the logs.

Comment 24 Alexey Tikhonov 2023-03-20 16:41:40 UTC
Upstream PR: https://github.com/SSSD/sssd/pull/6632

Comment 26 Alexey Tikhonov 2023-03-27 11:47:29 UTC
Pushed PR: https://github.com/SSSD/sssd/pull/6632

* `master`
    * 55e27a423d4065aa419e1bd80db1826eb8264c4a - KCM: Switch default caches only when there is no current default.
* `sssd-2-8`
    * 767234a3b216a19f94b3a41b15a444877c324494 - KCM: Switch default caches only when there is no current default.

Comment 34 errata-xmlrpc 2023-11-14 15:49:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7127