Bug 2016992 - Updating to 2.6.0 seems to break kdestroy/kinit/klist
Summary: Updating to 2.6.0 seems to break kdestroy/kinit/klist
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: sssd
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Alexey Tikhonov
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: sync-to-jira review
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-25 11:26 UTC by Ankur Sinha (FranciscoD)
Modified: 2021-11-05 01:08 UTC (History)
12 users (show)

Fixed In Version: sssd-2.6.0-2.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-05 01:08:13 UTC
Type: Bug


Attachments (Terms of Use)
/var/log/sssd/sssd_kcm.log (3.09 MB, text/plain)
2021-10-25 12:42 UTC, Ankur Sinha (FranciscoD)
no flags Details
new sssd_kcm.log file (3.12 MB, text/plain)
2021-10-25 15:22 UTC, Ankur Sinha (FranciscoD)
no flags Details
sssd_kcm log file from machine 2 (kdestroy on 2.5, upgrade, kinit on 2.6) (15.78 MB, text/plain)
2021-10-25 15:33 UTC, Ankur Sinha (FranciscoD)
no flags Details

Description Ankur Sinha (FranciscoD) 2021-10-25 11:26:13 UTC
Description of problem:
Updating to 2.6.0 seems to break kdestoy/kinit/klist

Version-Release number of selected component (if applicable):
2.6.0

How reproducible:
Always

Steps to Reproduce:
1. Update to 2.6.0 packages
2. Run `kdestroy` or `kinit` or `klist`
3.

Actual results:
Always get "Credentials cache I/O operation failed while resolving ccache"

Expected results:
Should work.

Additional info:
Here's the output from a fresh upgrade:


(ins)[asinha@ankur  ~]$ klist
Ticket cache: KCM:1000
Default principal: ankursinha

Valid starting     Expires            Service principal
25/10/21 12:15:07  26/10/21 12:15:07  krbtgt/FEDORAPROJECT.ORG
        renew until 01/11/21 11:15:07
(ins)[asinha@ankur  ~]$ sudo dnf update sssd\*
[sudo] password for asinha:
Fedora 35 - x86_64 - Updates                                                                             95 kB/s |  25 kB     00:00
Fedora Modular 35 - x86_64 - Updates                                                                    161 kB/s |  25 kB     00:00
Fedora 35 - x86_64 - Test Updates                                                                       237 kB/s |  16 kB     00:00
Fedora 35 - x86_64 - Test Updates                                                                       1.3 MB/s | 1.9 MB     00:01
Dependencies resolved.
========================================================================================================================================
 Package                             Architecture              Version                         Repository                          Size
========================================================================================================================================
Upgrading:
 libipa_hbac                         x86_64                    2.6.0-1.fc35                    updates-testing                     33 k
 libsss_certmap                      x86_64                    2.6.0-1.fc35                    updates-testing                     76 k
 libsss_idmap                        x86_64                    2.6.0-1.fc35                    updates-testing                     39 k
 libsss_nss_idmap                    x86_64                    2.6.0-1.fc35                    updates-testing                     41 k
 sssd                                x86_64                    2.6.0-1.fc35                    updates-testing                     25 k
 sssd-ad                             x86_64                    2.6.0-1.fc35                    updates-testing                    203 k
 sssd-client                         x86_64                    2.6.0-1.fc35                    updates-testing                    141 k
 sssd-common                         x86_64                    2.6.0-1.fc35                    updates-testing                    1.5 M
 sssd-common-pac                     x86_64                    2.6.0-1.fc35                    updates-testing                     89 k
 sssd-ipa                            x86_64                    2.6.0-1.fc35                    updates-testing                    269 k
 sssd-kcm                            x86_64                    2.6.0-1.fc35                    updates-testing                    104 k
 sssd-krb5                           x86_64                    2.6.0-1.fc35                    updates-testing                     76 k
 sssd-krb5-common                    x86_64                    2.6.0-1.fc35                    updates-testing                     82 k
 sssd-ldap                           x86_64                    2.6.0-1.fc35                    updates-testing                    153 k
 sssd-nfs-idmap                      x86_64                    2.6.0-1.fc35                    updates-testing                     36 k
 sssd-proxy                          x86_64                    2.6.0-1.fc35                    updates-testing                     66 k

Transaction Summary
========================================================================================================================================
Upgrade  16 Packages

Total download size: 2.9 M
Is this ok [y/N]: y
Downloading Packages:
(1/16): libipa_hbac-2.6.0-1.fc35.x86_64.rpm                                                             396 kB/s |  33 kB     00:00
(2/16): libsss_idmap-2.6.0-1.fc35.x86_64.rpm                                                            439 kB/s |  39 kB     00:00
(3/16): libsss_certmap-2.6.0-1.fc35.x86_64.rpm                                                          786 kB/s |  76 kB     00:00
(4/16): libsss_nss_idmap-2.6.0-1.fc35.x86_64.rpm                                                        1.2 MB/s |  41 kB     00:00
(5/16): sssd-2.6.0-1.fc35.x86_64.rpm                                                                    730 kB/s |  25 kB     00:00
(6/16): sssd-client-2.6.0-1.fc35.x86_64.rpm                                                             1.6 MB/s | 141 kB     00:00
(7/16): sssd-ad-2.6.0-1.fc35.x86_64.rpm                                                                 1.7 MB/s | 203 kB     00:00
(8/16): sssd-common-pac-2.6.0-1.fc35.x86_64.rpm                                                         1.8 MB/s |  89 kB     00:00
(9/16): sssd-kcm-2.6.0-1.fc35.x86_64.rpm                                                                2.0 MB/s | 104 kB     00:00
(10/16): sssd-krb5-2.6.0-1.fc35.x86_64.rpm                                                              1.3 MB/s |  76 kB     00:00
(11/16): sssd-krb5-common-2.6.0-1.fc35.x86_64.rpm                                                       1.7 MB/s |  82 kB     00:00
(12/16): sssd-ipa-2.6.0-1.fc35.x86_64.rpm                                                               1.2 MB/s | 269 kB     00:00
(13/16): sssd-nfs-idmap-2.6.0-1.fc35.x86_64.rpm                                                         725 kB/s |  36 kB     00:00
(14/16): sssd-proxy-2.6.0-1.fc35.x86_64.rpm                                                             968 kB/s |  66 kB     00:00
(15/16): sssd-ldap-2.6.0-1.fc35.x86_64.rpm                                                              508 kB/s | 153 kB     00:00
(16/16): sssd-common-2.6.0-1.fc35.x86_64.rpm                                                            1.4 MB/s | 1.5 MB     00:01
----------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                   2.3 MB/s | 2.9 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                1/1
  Upgrading        : libsss_idmap-2.6.0-1.fc35.x86_64                                                                              1/32
  Upgrading        : libsss_certmap-2.6.0-1.fc35.x86_64                                                                            2/32
  Upgrading        : sssd-nfs-idmap-2.6.0-1.fc35.x86_64                                                                            3/32
  Upgrading        : libsss_nss_idmap-2.6.0-1.fc35.x86_64                                                                          4/32
  Upgrading        : sssd-client-2.6.0-1.fc35.x86_64                                                                               5/32
  Running scriptlet: sssd-client-2.6.0-1.fc35.x86_64                                                                               5/32
  Upgrading        : sssd-common-2.6.0-1.fc35.x86_64                                                                               6/32
  Running scriptlet: sssd-common-2.6.0-1.fc35.x86_64                                                                               6/32
  Upgrading        : sssd-krb5-common-2.6.0-1.fc35.x86_64                                                                          7/32
  Upgrading        : sssd-common-pac-2.6.0-1.fc35.x86_64                                                                           8/32
  Upgrading        : sssd-ad-2.6.0-1.fc35.x86_64                                                                                   9/32
  Upgrading        : sssd-krb5-2.6.0-1.fc35.x86_64                                                                                10/32
  Upgrading        : sssd-ldap-2.6.0-1.fc35.x86_64                                                                                11/32
  Upgrading        : sssd-proxy-2.6.0-1.fc35.x86_64                                                                               12/32
  Upgrading        : libipa_hbac-2.6.0-1.fc35.x86_64                                                                              13/32
  Upgrading        : sssd-ipa-2.6.0-1.fc35.x86_64                                                                                 14/32
  Upgrading        : sssd-2.6.0-1.fc35.x86_64                                                                                     15/32
  Upgrading        : sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 16/32
  Running scriptlet: sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 16/32
  Cleanup          : sssd-2.5.2-5.fc35.x86_64                                                                                     17/32
  Cleanup          : sssd-ipa-2.5.2-5.fc35.x86_64                                                                                 18/32
  Cleanup          : sssd-ad-2.5.2-5.fc35.x86_64                                                                                  19/32
  Cleanup          : sssd-ldap-2.5.2-5.fc35.x86_64                                                                                20/32
  Cleanup          : sssd-common-pac-2.5.2-5.fc35.x86_64                                                                          21/32
  Cleanup          : sssd-krb5-2.5.2-5.fc35.x86_64                                                                                22/32
  Cleanup          : sssd-proxy-2.5.2-5.fc35.x86_64                                                                               23/32
  Running scriptlet: sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 24/32
  Cleanup          : sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 24/32
  Running scriptlet: sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 24/32
  Cleanup          : sssd-krb5-common-2.5.2-5.fc35.x86_64                                                                         25/32
  Running scriptlet: sssd-common-2.5.2-5.fc35.x86_64                                                                              26/32
  Cleanup          : sssd-common-2.5.2-5.fc35.x86_64                                                                              26/32
  Running scriptlet: sssd-common-2.5.2-5.fc35.x86_64                                                                              26/32
  Running scriptlet: sssd-client-2.5.2-5.fc35.x86_64                                                                              27/32
  Cleanup          : sssd-client-2.5.2-5.fc35.x86_64                                                                              27/32
  Cleanup          : libsss_idmap-2.5.2-5.fc35.x86_64                                                                             28/32
  Cleanup          : libsss_nss_idmap-2.5.2-5.fc35.x86_64                                                                         29/32
  Cleanup          : libsss_certmap-2.5.2-5.fc35.x86_64                                                                           30/32
  Cleanup          : sssd-nfs-idmap-2.5.2-5.fc35.x86_64                                                                           31/32
  Cleanup          : libipa_hbac-2.5.2-5.fc35.x86_64                                                                              32/32
  Running scriptlet: sssd-common-2.6.0-1.fc35.x86_64                                                                              32/32
  Running scriptlet: libipa_hbac-2.5.2-5.fc35.x86_64                                                                              32/32
  Verifying        : libipa_hbac-2.6.0-1.fc35.x86_64                                                                               1/32
  Verifying        : libipa_hbac-2.5.2-5.fc35.x86_64                                                                               2/32
  Verifying        : libsss_certmap-2.6.0-1.fc35.x86_64                                                                            3/32
  Verifying        : libsss_certmap-2.5.2-5.fc35.x86_64                                                                            4/32
  Verifying        : libsss_idmap-2.6.0-1.fc35.x86_64                                                                              5/32
  Verifying        : libsss_idmap-2.5.2-5.fc35.x86_64                                                                              6/32
  Verifying        : libsss_nss_idmap-2.6.0-1.fc35.x86_64                                                                          7/32
  Verifying        : libsss_nss_idmap-2.5.2-5.fc35.x86_64                                                                          8/32
  Verifying        : sssd-2.6.0-1.fc35.x86_64                                                                                      9/32
  Verifying        : sssd-2.5.2-5.fc35.x86_64                                                                                     10/32
  Verifying        : sssd-ad-2.6.0-1.fc35.x86_64                                                                                  11/32
  Verifying        : sssd-ad-2.5.2-5.fc35.x86_64                                                                                  12/32
  Verifying        : sssd-client-2.6.0-1.fc35.x86_64                                                                              13/32
  Verifying        : sssd-client-2.5.2-5.fc35.x86_64                                                                              14/32
  Verifying        : sssd-common-2.6.0-1.fc35.x86_64                                                                              15/32
  Verifying        : sssd-common-2.5.2-5.fc35.x86_64                                                                              16/32
  Verifying        : sssd-common-pac-2.6.0-1.fc35.x86_64                                                                          17/32
  Verifying        : sssd-common-pac-2.5.2-5.fc35.x86_64                                                                          18/32
  Verifying        : sssd-ipa-2.6.0-1.fc35.x86_64                                                                                 19/32
  Verifying        : sssd-ipa-2.5.2-5.fc35.x86_64                                                                                 20/32
  Verifying        : sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 21/32
  Verifying        : sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 22/32
  Verifying        : sssd-krb5-2.6.0-1.fc35.x86_64                                                                                23/32
  Verifying        : sssd-krb5-2.5.2-5.fc35.x86_64                                                                                24/32
  Verifying        : sssd-krb5-common-2.6.0-1.fc35.x86_64                                                                         25/32
  Verifying        : sssd-krb5-common-2.5.2-5.fc35.x86_64                                                                         26/32
  Verifying        : sssd-ldap-2.6.0-1.fc35.x86_64                                                                                27/32
  Verifying        : sssd-ldap-2.5.2-5.fc35.x86_64                                                                                28/32
  Verifying        : sssd-nfs-idmap-2.6.0-1.fc35.x86_64                                                                           29/32
  Verifying        : sssd-nfs-idmap-2.5.2-5.fc35.x86_64                                                                           30/32
  Verifying        : sssd-proxy-2.6.0-1.fc35.x86_64                                                                               31/32
  Verifying        : sssd-proxy-2.5.2-5.fc35.x86_64                                                                               32/32
You should restart:
  * Some applications using:
      dropbox stop; dropbox start
      killall -3 gnome-shell
      sudo systemctl restart NetworkManager
      sudo systemctl restart abrt-journal-core
      sudo systemctl restart abrt-oops
      sudo systemctl restart abrt-xorg
      sudo systemctl restart abrtd
      sudo systemctl restart accounts-daemon
      sudo systemctl restart atd
      sudo systemctl restart auditd
      sudo systemctl restart avahi-daemon
      sudo systemctl restart chronyd
      sudo systemctl restart colord
      sudo systemctl restart crond
      sudo systemctl restart cups
      sudo systemctl restart dbus-:1.16-org.freedesktop.problems@0
      sudo systemctl restart dbus-broker
      sudo systemctl restart firewalld
      sudo systemctl restart gdm
      sudo systemctl restart libvirtd
      sudo systemctl restart mcelog
      sudo systemctl restart packagekit
      sudo systemctl restart polkit
      sudo systemctl restart power-profiles-daemon
      sudo systemctl restart rtkit-daemon
      sudo systemctl restart sshd
      sudo systemctl restart sssd-kcm
      sudo systemctl restart systemd-logind
      sudo systemctl restart systemd-resolved
      sudo systemctl restart systemd-udevd
      sudo systemctl restart udisks2

  * These applications manually:
      (sd-pam)
      AuthManagerDaemon
      Xorg
      deja-dup
      dnf
      evolution-alarm-notify
      evolution-calendar-factory
      evolution-source-registry
      fusermount
      gdm-session-worker
      gnome-session-binary
      gnome-software
      goa-identity-service
      gsd-media-keys
      gsd-power
      ibus-daemon
      icasessionmgr
      pipewire
      pipewire-pulse
      qutebrowser
      wireplumber
      xdg-desktop-portal
      xdg-desktop-portal-gnome
      zoom

Additionally, there are:
  - 3 processes requiring restart of your session (i.e. Logging out & Logging in again)

For more information run:
    sudo tracer -iat 1635160525.9518569

Upgraded:
  libipa_hbac-2.6.0-1.fc35.x86_64               libsss_certmap-2.6.0-1.fc35.x86_64          libsss_idmap-2.6.0-1.fc35.x86_64
  libsss_nss_idmap-2.6.0-1.fc35.x86_64          sssd-2.6.0-1.fc35.x86_64                    sssd-ad-2.6.0-1.fc35.x86_64
  sssd-client-2.6.0-1.fc35.x86_64               sssd-common-2.6.0-1.fc35.x86_64             sssd-common-pac-2.6.0-1.fc35.x86_64
  sssd-ipa-2.6.0-1.fc35.x86_64                  sssd-kcm-2.6.0-1.fc35.x86_64                sssd-krb5-2.6.0-1.fc35.x86_64
  sssd-krb5-common-2.6.0-1.fc35.x86_64          sssd-ldap-2.6.0-1.fc35.x86_64               sssd-nfs-idmap-2.6.0-1.fc35.x86_64
  sssd-proxy-2.6.0-1.fc35.x86_64

Complete!

(ins)[asinha@ankur  ~]$ klist
klist: Credentials cache I/O operation failed while resolving ccache

(ins)[asinha@ankur  ~]$ kdestroy && knit
kdestroy: Credentials cache I/O operation failed while resolving ccache

(ins)[asinha@ankur  ~]$ export KRB5_TRACE=/dev/stderr
(ins)[asinha@ankur  ~]$ kdestroy && knit
kdestroy: Credentials cache I/O operation failed while resolving ccache

(ins)[asinha@ankur  ~]$ kdestroy && knit
kdestroy: Credentials cache I/O operation failed while resolving ccache

(ins)[asinha@ankur  ~]$ kdestroy && knit
kdestroy: Credentials cache I/O operation failed while resolving ccache

(ins)[asinha@ankur  ~]$ ls /tmp/
hsperfdata_asinha
qtsingleapp-zoom-3e8
qtsingleapp-zoom-3e8-lockfile
ssh-XXXXXX68pTNV
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-bluetooth.service-NunzrQ
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-chronyd.service-xAItvY
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-colord.service-Yg5n5c
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-dbus-broker.service-MgWgnx
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-geoclue.service-brjUBc
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-low-memory-monitor.service-kwf3E4
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-ModemManager.service-9oZuVw
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-power-profiles-daemon.service-MB8Qbq
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-rtkit-daemon.service-FvhCRd
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-switcheroo-control.service-pBS8Gh
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-systemd-logind.service-XddtLu
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-systemd-oomd.service-Apy8Uq
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-systemd-resolved.service-RCxzv5
systemd-private-b71a2c6e3a724e0f8ae3b8403d6a99d7-upower.service-uo9cFd
tmux-1000
tracker-extract-3-files.1000
(ins)[asinha@ankur  ~]$

(ins)[asinha@ankur  ~]$ sudo systemctl restart sssd*
[sudo] password for asinha: 

(ins)[asinha@ankur  ~]$ kdestroy && knit
kdestroy: Credentials cache I/O operation failed while resolving ccache


----

I'd used `dnf offline-upgrade` the first time and had seen these issues after the reboot also.

I'm not well versed enough with these bits to provide more logs. If you can please point me to what logs I can provide, I'll attach them too.

Thanks very much,

Comment 1 Ankur Sinha (FranciscoD) 2021-10-25 11:27:21 UTC
Please also note: these systems are otherwise up to date F35 systems with updates-testing enabled, upgraded from F34.

Comment 2 Ankur Sinha (FranciscoD) 2021-10-25 11:30:31 UTC
Here's after undoing the transaction to go back down to 2.5.2:


(ins)[asinha@ankur  ~]$ sudo dnf history undo last -y
RPM Fusion for Fedora 35 - Free                                                                          26 kB/s | 3.1 kB     00:00
RPM Fusion for Fedora 35 - Free - Test Updates                                                           39 kB/s | 4.3 kB     00:00
RPM Fusion for Fedora 35 - Nonfree                                                                       31 kB/s | 2.9 kB     00:00
RPM Fusion for Fedora 35 - Nonfree - Test Updates                                                        34 kB/s | 4.1 kB     00:00
Dependencies resolved.
========================================================================================================================================
 Package                               Architecture                Version                            Repository                   Size
========================================================================================================================================
Downgrading:
 libipa_hbac                           x86_64                      2.5.2-5.fc35                       fedora                       34 k
 libsss_certmap                        x86_64                      2.5.2-5.fc35                       fedora                       70 k
 libsss_idmap                          x86_64                      2.5.2-5.fc35                       fedora                       39 k
 libsss_nss_idmap                      x86_64                      2.5.2-5.fc35                       fedora                       41 k
 sssd                                  x86_64                      2.5.2-5.fc35                       fedora                       25 k
 sssd-ad                               x86_64                      2.5.2-5.fc35                       fedora                      183 k
 sssd-client                           x86_64                      2.5.2-5.fc35                       fedora                      127 k
 sssd-common                           x86_64                      2.5.2-5.fc35                       fedora                      1.5 M
 sssd-common-pac                       x86_64                      2.5.2-5.fc35                       fedora                       90 k
 sssd-ipa                              x86_64                      2.5.2-5.fc35                       fedora                      260 k
 sssd-kcm                              x86_64                      2.5.2-5.fc35                       fedora                      109 k
 sssd-krb5                             x86_64                      2.5.2-5.fc35                       fedora                       68 k
 sssd-krb5-common                      x86_64                      2.5.2-5.fc35                       fedora                       82 k
 sssd-ldap                             x86_64                      2.5.2-5.fc35                       fedora                      127 k
 sssd-nfs-idmap                        x86_64                      2.5.2-5.fc35                       fedora                       35 k
 sssd-proxy                            x86_64                      2.5.2-5.fc35                       fedora                       67 k

Transaction Summary
========================================================================================================================================
Downgrade  16 Packages

Total download size: 2.8 M
Downloading Packages:
(1/16): libsss_idmap-2.5.2-5.fc35.x86_64.rpm                                                            252 kB/s |  39 kB     00:00
(2/16): libsss_certmap-2.5.2-5.fc35.x86_64.rpm                                                          433 kB/s |  70 kB     00:00
(3/16): libipa_hbac-2.5.2-5.fc35.x86_64.rpm                                                             201 kB/s |  34 kB     00:00
(4/16): libsss_nss_idmap-2.5.2-5.fc35.x86_64.rpm                                                        713 kB/s |  41 kB     00:00
(5/16): sssd-2.5.2-5.fc35.x86_64.rpm                                                                    232 kB/s |  25 kB     00:00
(6/16): sssd-ad-2.5.2-5.fc35.x86_64.rpm                                                                 1.4 MB/s | 183 kB     00:00
(7/16): sssd-client-2.5.2-5.fc35.x86_64.rpm                                                             1.1 MB/s | 127 kB     00:00
(8/16): sssd-common-pac-2.5.2-5.fc35.x86_64.rpm                                                         1.7 MB/s |  90 kB     00:00
(9/16): sssd-ipa-2.5.2-5.fc35.x86_64.rpm                                                                1.4 MB/s | 260 kB     00:00
(10/16): sssd-kcm-2.5.2-5.fc35.x86_64.rpm                                                               557 kB/s | 109 kB     00:00
(11/16): sssd-krb5-2.5.2-5.fc35.x86_64.rpm                                                              482 kB/s |  68 kB     00:00
(12/16): sssd-krb5-common-2.5.2-5.fc35.x86_64.rpm                                                       635 kB/s |  82 kB     00:00
(13/16): sssd-nfs-idmap-2.5.2-5.fc35.x86_64.rpm                                                         317 kB/s |  35 kB     00:00
(14/16): sssd-ldap-2.5.2-5.fc35.x86_64.rpm                                                              817 kB/s | 127 kB     00:00
(15/16): sssd-proxy-2.5.2-5.fc35.x86_64.rpm                                                             808 kB/s |  67 kB     00:00
(16/16): sssd-common-2.5.2-5.fc35.x86_64.rpm                                                            2.4 MB/s | 1.5 MB     00:00
----------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                   2.8 MB/s | 2.8 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                1/1
  Downgrading      : libsss_idmap-2.5.2-5.fc35.x86_64                                                                              1/32
  Downgrading      : libsss_certmap-2.5.2-5.fc35.x86_64                                                                            2/32
  Downgrading      : sssd-nfs-idmap-2.5.2-5.fc35.x86_64                                                                            3/32
  Downgrading      : libsss_nss_idmap-2.5.2-5.fc35.x86_64                                                                          4/32
  Downgrading      : sssd-client-2.5.2-5.fc35.x86_64                                                                               5/32
  Running scriptlet: sssd-client-2.5.2-5.fc35.x86_64                                                                               5/32
  Downgrading      : sssd-common-2.5.2-5.fc35.x86_64                                                                               6/32
  Running scriptlet: sssd-common-2.5.2-5.fc35.x86_64                                                                               6/32
  Downgrading      : sssd-krb5-common-2.5.2-5.fc35.x86_64                                                                          7/32
  Downgrading      : sssd-common-pac-2.5.2-5.fc35.x86_64                                                                           8/32
  Downgrading      : sssd-ad-2.5.2-5.fc35.x86_64                                                                                   9/32
  Downgrading      : sssd-krb5-2.5.2-5.fc35.x86_64                                                                                10/32
  Downgrading      : sssd-ldap-2.5.2-5.fc35.x86_64                                                                                11/32
  Downgrading      : sssd-proxy-2.5.2-5.fc35.x86_64                                                                               12/32
  Downgrading      : libipa_hbac-2.5.2-5.fc35.x86_64                                                                              13/32
  Downgrading      : sssd-ipa-2.5.2-5.fc35.x86_64                                                                                 14/32
  Downgrading      : sssd-2.5.2-5.fc35.x86_64                                                                                     15/32
  Downgrading      : sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 16/32
  Running scriptlet: sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 16/32
  Cleanup          : sssd-2.6.0-1.fc35.x86_64                                                                                     17/32
  Cleanup          : sssd-ipa-2.6.0-1.fc35.x86_64                                                                                 18/32
  Cleanup          : sssd-ad-2.6.0-1.fc35.x86_64                                                                                  19/32
  Cleanup          : sssd-ldap-2.6.0-1.fc35.x86_64                                                                                20/32
  Cleanup          : sssd-common-pac-2.6.0-1.fc35.x86_64                                                                          21/32
  Cleanup          : sssd-krb5-2.6.0-1.fc35.x86_64                                                                                22/32
  Cleanup          : sssd-proxy-2.6.0-1.fc35.x86_64                                                                               23/32
  Running scriptlet: sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 24/32
  Cleanup          : sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 24/32
  Running scriptlet: sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 24/32
  Cleanup          : sssd-krb5-common-2.6.0-1.fc35.x86_64                                                                         25/32
  Running scriptlet: sssd-common-2.6.0-1.fc35.x86_64                                                                              26/32
  Cleanup          : sssd-common-2.6.0-1.fc35.x86_64                                                                              26/32
  Running scriptlet: sssd-common-2.6.0-1.fc35.x86_64                                                                              26/32
  Running scriptlet: sssd-client-2.6.0-1.fc35.x86_64                                                                              27/32
  Cleanup          : sssd-client-2.6.0-1.fc35.x86_64                                                                              27/32
  Cleanup          : libsss_idmap-2.6.0-1.fc35.x86_64                                                                             28/32
  Cleanup          : libsss_nss_idmap-2.6.0-1.fc35.x86_64                                                                         29/32
  Cleanup          : libsss_certmap-2.6.0-1.fc35.x86_64                                                                           30/32
  Cleanup          : sssd-nfs-idmap-2.6.0-1.fc35.x86_64                                                                           31/32
  Cleanup          : libipa_hbac-2.6.0-1.fc35.x86_64                                                                              32/32
  Running scriptlet: sssd-common-2.5.2-5.fc35.x86_64                                                                              32/32
  Running scriptlet: libipa_hbac-2.6.0-1.fc35.x86_64                                                                              32/32
  Verifying        : libipa_hbac-2.5.2-5.fc35.x86_64                                                                               1/32
  Verifying        : libipa_hbac-2.6.0-1.fc35.x86_64                                                                               2/32
  Verifying        : libsss_certmap-2.5.2-5.fc35.x86_64                                                                            3/32
  Verifying        : libsss_certmap-2.6.0-1.fc35.x86_64                                                                            4/32
  Verifying        : libsss_idmap-2.5.2-5.fc35.x86_64                                                                              5/32
  Verifying        : libsss_idmap-2.6.0-1.fc35.x86_64                                                                              6/32
  Verifying        : libsss_nss_idmap-2.5.2-5.fc35.x86_64                                                                          7/32
  Verifying        : libsss_nss_idmap-2.6.0-1.fc35.x86_64                                                                          8/32
  Verifying        : sssd-2.5.2-5.fc35.x86_64                                                                                      9/32
  Verifying        : sssd-2.6.0-1.fc35.x86_64                                                                                     10/32
  Verifying        : sssd-ad-2.5.2-5.fc35.x86_64                                                                                  11/32
  Verifying        : sssd-ad-2.6.0-1.fc35.x86_64                                                                                  12/32
  Verifying        : sssd-client-2.5.2-5.fc35.x86_64                                                                              13/32
  Verifying        : sssd-client-2.6.0-1.fc35.x86_64                                                                              14/32
  Verifying        : sssd-common-2.5.2-5.fc35.x86_64                                                                              15/32
  Verifying        : sssd-common-2.6.0-1.fc35.x86_64                                                                              16/32
  Verifying        : sssd-common-pac-2.5.2-5.fc35.x86_64                                                                          17/32
  Verifying        : sssd-common-pac-2.6.0-1.fc35.x86_64                                                                          18/32
  Verifying        : sssd-ipa-2.5.2-5.fc35.x86_64                                                                                 19/32
  Verifying        : sssd-ipa-2.6.0-1.fc35.x86_64                                                                                 20/32
  Verifying        : sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 21/32
  Verifying        : sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 22/32
  Verifying        : sssd-krb5-2.5.2-5.fc35.x86_64                                                                                23/32
  Verifying        : sssd-krb5-2.6.0-1.fc35.x86_64                                                                                24/32
  Verifying        : sssd-krb5-common-2.5.2-5.fc35.x86_64                                                                         25/32
  Verifying        : sssd-krb5-common-2.6.0-1.fc35.x86_64                                                                         26/32
  Verifying        : sssd-ldap-2.5.2-5.fc35.x86_64                                                                                27/32
  Verifying        : sssd-ldap-2.6.0-1.fc35.x86_64                                                                                28/32
  Verifying        : sssd-nfs-idmap-2.5.2-5.fc35.x86_64                                                                           29/32
  Verifying        : sssd-nfs-idmap-2.6.0-1.fc35.x86_64                                                                           30/32
  Verifying        : sssd-proxy-2.5.2-5.fc35.x86_64                                                                               31/32
  Verifying        : sssd-proxy-2.6.0-1.fc35.x86_64                                                                               32/32
You should restart:
  * Some applications using:
      dropbox stop; dropbox start
      killall -3 gnome-shell
      sudo systemctl restart NetworkManager
      sudo systemctl restart abrt-journal-core
      sudo systemctl restart abrt-oops
      sudo systemctl restart abrt-xorg
      sudo systemctl restart abrtd
      sudo systemctl restart accounts-daemon
      sudo systemctl restart atd
      sudo systemctl restart auditd
      sudo systemctl restart avahi-daemon
      sudo systemctl restart chronyd
      sudo systemctl restart colord
      sudo systemctl restart crond
      sudo systemctl restart cups
      sudo systemctl restart dbus-:1.16-org.freedesktop.problems@0
      sudo systemctl restart dbus-broker
      sudo systemctl restart firewalld
      sudo systemctl restart gdm
      sudo systemctl restart libvirtd
      sudo systemctl restart mcelog
      sudo systemctl restart packagekit
      sudo systemctl restart polkit
      sudo systemctl restart power-profiles-daemon
      sudo systemctl restart rtkit-daemon
      sudo systemctl restart sshd
      sudo systemctl restart sssd-kcm
      sudo systemctl restart systemd-logind
      sudo systemctl restart systemd-resolved
      sudo systemctl restart systemd-udevd
      sudo systemctl restart udisks2

  * These applications manually:
      (sd-pam)
      AuthManagerDaemon
      Xorg
      deja-dup
      dnf
      evolution-alarm-notify
      evolution-calendar-factory
      evolution-source-registry
      fusermount
      gdm-session-worker
      gnome-session-binary
      gnome-software
      goa-identity-service
      gsd-media-keys
      gsd-power
      ibus-daemon
      icasessionmgr
      pipewire
      pipewire-pulse
      qutebrowser
      wireplumber
      xdg-desktop-portal
      xdg-desktop-portal-gnome
      zoom

Additionally, there are:
  - 3 processes requiring restart of your session (i.e. Logging out & Logging in again)

For more information run:
    sudo tracer -iat 1635161263.1342258

Downgraded:
  libipa_hbac-2.5.2-5.fc35.x86_64               libsss_certmap-2.5.2-5.fc35.x86_64          libsss_idmap-2.5.2-5.fc35.x86_64
  libsss_nss_idmap-2.5.2-5.fc35.x86_64          sssd-2.5.2-5.fc35.x86_64                    sssd-ad-2.5.2-5.fc35.x86_64
  sssd-client-2.5.2-5.fc35.x86_64               sssd-common-2.5.2-5.fc35.x86_64             sssd-common-pac-2.5.2-5.fc35.x86_64
  sssd-ipa-2.5.2-5.fc35.x86_64                  sssd-kcm-2.5.2-5.fc35.x86_64                sssd-krb5-2.5.2-5.fc35.x86_64
  sssd-krb5-common-2.5.2-5.fc35.x86_64          sssd-ldap-2.5.2-5.fc35.x86_64               sssd-nfs-idmap-2.5.2-5.fc35.x86_64
  sssd-proxy-2.5.2-5.fc35.x86_64

Complete!

(ins)[asinha@ankur  ~]$ sudo systemctl restart sssd*

(ins)[asinha@ankur  ~]$ klist
Ticket cache: KCM:1000
Default principal: ankursinha

Valid starting     Expires            Service principal
25/10/21 12:15:07  26/10/21 12:15:07  krbtgt/FEDORAPROJECT.ORG
        renew until 01/11/21 11:15:07

(ins)[asinha@ankur  ~]$

Comment 3 Alexey Tikhonov 2021-10-25 11:40:30 UTC
Hi,

could you please set (add)
```
[kcm]
debug_level = 9
```
to `/etc/sssd/sssd.conf` (if you will create this file from scratch, please take a note it should be owned by root:root; or you can use config snippet, see `man sssd.conf`:"CONFIGURATION SNIPPETS")

repeat `date; klist` (with 2.6.0) and provide output and /var/log/sssd/sssd_kcm.log?

Comment 4 Ankur Sinha (FranciscoD) 2021-10-25 12:42:32 UTC
Created attachment 1836735 [details]
/var/log/sssd/sssd_kcm.log

Hi,

here's the output:

(ins)[asinha@ankur  ~]$ sudo ls -lash /etc/sssd/
total 28K
4.0K drwx------.   4 root root 4.0K Oct 25 13:33 .
 12K drwxr-xr-x. 191 root root  12K Oct 25 13:31 ..
4.0K drwx--x--x.   2 root root 4.0K Oct 14 12:31 conf.d
4.0K drwx--x--x.   2 root root 4.0K Oct 14 12:31 pki
4.0K -rw-r--r--.   1 root root   22 Oct 25 13:33 sssd.conf

(ins)[asinha@ankur  ~]$ sudo cat /etc/sssd/sssd.conf
[kcm]
debug_level = 9

# sudo systemctl restart sssd*

(ins)[asinha@ankur  ~]$ date ; klist
Mon 25 Oct 13:37:42 BST 2021
klist: Credentials cache I/O operation failed while resolving ccache


The log file is attached.

Comment 5 Alexey Tikhonov 2021-10-25 13:44:20 UTC
Hi,

(In reply to Ankur Sinha (FranciscoD) from comment #4)
> Created attachment 1836735 [details]
> /var/log/sssd/sssd_kcm.log

Thank you.
I forgot to mention it's required to restart sssd_kcm for setting to take effect, but actually log contains backtrace even with default setting, so that's useful.


It would be great to see output of:
```
# ldbsearch -H /var/lib/sss/secrets/secrets.ldb 'dn=cn=default,cn=1000,cn=persistent,cn=kcm'
```

Take a note: secrets.ldb contains all ccaches, i.e. includes sensitive data.
This specific record - 'dn=cn=default,cn=1000,cn=persistent,cn=kcm' - is NOT expected to contain sensitive data (only randomly generated UUID of actual ccache) but, since in your case it contains something not expected, to be on a safe side you might want to email output directly to my email atikhono

Comment 6 Alexey Tikhonov 2021-10-25 13:53:05 UTC
I'm especially interested in `creationTime: ` of this entry.
I wonder if this is a couple of Fedora releases old...

Comment 7 Ankur Sinha (FranciscoD) 2021-10-25 14:13:46 UTC
(In reply to Alexey Tikhonov from comment #5)
> Hi,
> 
> (In reply to Ankur Sinha (FranciscoD) from comment #4)
> > Created attachment 1836735 [details]
> > /var/log/sssd/sssd_kcm.log
> 
> Thank you.
> I forgot to mention it's required to restart sssd_kcm for setting to take
> effect, but actually log contains backtrace even with default setting, so
> that's useful.

I did restart all the systemd services: `systemctl restart sssd*`, so hopefully it also restarted sssd_kcm (maybe that's why it has the logs).

> 
> 
> It would be great to see output of:
> ```
> # ldbsearch -H /var/lib/sss/secrets/secrets.ldb
> 'dn=cn=default,cn=1000,cn=persistent,cn=kcm'
> ```
> 
> Take a note: secrets.ldb contains all ccaches, i.e. includes sensitive data.
> This specific record - 'dn=cn=default,cn=1000,cn=persistent,cn=kcm' - is NOT
> expected to contain sensitive data (only randomly generated UUID of actual
> ccache) but, since in your case it contains something not expected, to be on
> a safe side you might want to email output directly to my email
> atikhono

I've e-mailed the output to you now. 


(In reply to Alexey Tikhonov from comment #6)
> I'm especially interested in `creationTime: ` of this entry.
> I wonder if this is a couple of Fedora releases old...

This system hasn't been reinstalled in a while---I upgrade from release to release. I have the "Fedora Account" setup in Gnome Online Accounts also. 

Thanks very much,

Comment 8 Alexey Tikhonov 2021-10-25 14:32:57 UTC
(In reply to Ankur Sinha (FranciscoD) from comment #7)
> 
> I've e-mailed the output to you now. 
> 
> (In reply to Alexey Tikhonov from comment #6)
> > I'm especially interested in `creationTime: ` of this entry.
> > I wonder if this is a couple of Fedora releases old...
> 
> This system hasn't been reinstalled in a while

creationTime: 1558356650
enctype: masterkey

```
# date -d @1558356650
Mon 20 May 2019 02:50:50 PM CEST
```

Well, this is expected that so old credentials won't be supported starting 2.6.0 upstream release (SSSD doesn't create ccaches in this format since Fedora 33)
So you will have to re-obtain ticket.
But of course I meant to handle this more gracefully (moreover, I even tested this, but it seems it wasn't enough)...

Could you please check if `kdestroy` + `kinit` works for you with 2.6.0?

Comment 9 Alexey Tikhonov 2021-10-25 14:39:50 UTC
(In reply to Alexey Tikhonov from comment #8)
> 
> Could you please check if `kdestroy` + `kinit` works for you with 2.6.0?

Ah, sorry, you mentioned this:
```
$ kdestroy && knit
kdestroy: Credentials cache I/O operation failed while resolving ccache
```

Comment 10 Ankur Sinha (FranciscoD) 2021-10-25 14:47:23 UTC
Ah, sorry about that. I'm happy to nuke the old ticket and create a new one if there's a way to do that. Maybe I can downgrade, `kdestroy`, then upgrade and then `kinit`? Would that workaround the "credentials cache.." issue?

Comment 11 Pavel Březina 2021-10-25 15:08:20 UTC
I've tested upgrade path too, quite extensively so I'm curious how we missed this. Anyway it looks like ccdb_secdb_get_default_send() does not validate the iobuffer through sec_kv_to_ccache_binary() so it does not purge the cache.

It should be enough to delete /var/lib/sss/secrets/secrets.ldb and run kinit but if you don't mind, please create a copy of it and keep it for testing fix for this bug.

Comment 12 Alexey Tikhonov 2021-10-25 15:11:42 UTC
(In reply to Ankur Sinha (FranciscoD) from comment #10)
> Ah, sorry about that. I'm happy to nuke the old ticket and create a new one
> if there's a way to do that. Maybe I can downgrade, `kdestroy`, then upgrade
> and then `kinit`? Would that workaround the "credentials cache.." issue?

I think it should, but I'd like to ask you to keep you ccache intact until I was able to reproduce this locally.

But in your original email you wrote "just updated two F35 systems", so perhaps you could try this (kdestroy and kinit) on one of systems?
Moreover, I didn't actually see any failed attempt to `kinit` in sssd_kcm.log. What is the output of kinit?

Comment 13 Ankur Sinha (FranciscoD) 2021-10-25 15:22:45 UTC
Created attachment 1836909 [details]
new sssd_kcm.log file

(In reply to Pavel Březina from comment #11)
> I've tested upgrade path too, quite extensively so I'm curious how we missed
> this. Anyway it looks like ccdb_secdb_get_default_send() does not validate
> the iobuffer through sec_kv_to_ccache_binary() so it does not purge the
> cache.
> 
> It should be enough to delete /var/lib/sss/secrets/secrets.ldb and run kinit
> but if you don't mind, please create a copy of it and keep it for testing
> fix for this bug.

Sure. I've made a copy of this already and will keep it safe.

(In reply to Alexey Tikhonov from comment #12)
> (In reply to Ankur Sinha (FranciscoD) from comment #10)
> > Ah, sorry about that. I'm happy to nuke the old ticket and create a new one
> > if there's a way to do that. Maybe I can downgrade, `kdestroy`, then upgrade
> > and then `kinit`? Would that workaround the "credentials cache.." issue?
> 
> I think it should, but I'd like to ask you to keep you ccache intact until I
> was able to reproduce this locally.
> 
> But in your original email you wrote "just updated two F35 systems", so
> perhaps you could try this (kdestroy and kinit) on one of systems?

Sure. I downgraded the other system to be able to build packages etc., I can kdestroy there, upgrade to 2.6.0 and try kinit to see what happens and report back.


> Moreover, I didn't actually see any failed attempt to `kinit` in
> sssd_kcm.log. What is the output of kinit?

I get the same thing with all kdestroy/klist/kinit:

(ins)[asinha@ankur  ~]$ date
Mon 25 Oct 16:20:31 BST 2021
(ins)[asinha@ankur  ~]$ klist
klist: Credentials cache I/O operation failed while resolving ccache
(ins)[asinha@ankur  ~]$ kinit
kinit: Credentials cache I/O operation failed while getting default ccache
(ins)[asinha@ankur  ~]$ kdestroy
kdestroy: Credentials cache I/O operation failed while resolving ccache]

sssd_kcm.log attached again.

Comment 14 Ankur Sinha (FranciscoD) 2021-10-25 15:33:08 UTC
Created attachment 1836922 [details]
sssd_kcm log file from machine 2 (kdestroy on 2.5, upgrade, kinit on 2.6)

On the second machine: kdestroy on 2.5, upgrade, and then kinit does not work, unfortunately:


(ins)[asinha@ankur  ~]$ klist
Ticket cache: KCM:1000
Default principal: ankursinha

Valid starting     Expires            Service principal
25/10/21 09:01:35  26/10/21 07:56:18  HTTP/koji.fedoraproject.org@
        renew until 01/11/21 06:56:18
        Ticket server: HTTP/koji.fedoraproject.org
25/10/21 07:56:18  26/10/21 07:56:18  krbtgt/FEDORAPROJECT.ORG
        renew until 01/11/21 06:56:18
25/10/21 12:21:57  26/10/21 07:56:18  HTTP/id.fedoraproject.org@
        renew until 01/11/21 06:56:18
        Ticket server: HTTP/id.fedoraproject.org

(ins)[asinha@ankur  ~]$ kdestroy

(ins)[asinha@ankur  ~]$ klist
klist: Credentials cache 'KCM:1000' not found

(ins)[asinha@ankur  ~]$ sudo dnf update sssd\*
[sudo] password for asinha:
Last metadata expiration check: 0:46:30 ago on Mon 25 Oct 2021 15:36:31 BST.
Dependencies resolved.
========================================================================================================================================
 Package                             Architecture              Version                         Repository                          Size
========================================================================================================================================
Upgrading:
 libipa_hbac                         x86_64                    2.6.0-1.fc35                    updates-testing                     33 k
 libsss_certmap                      x86_64                    2.6.0-1.fc35                    updates-testing                     76 k
 libsss_idmap                        x86_64                    2.6.0-1.fc35                    updates-testing                     39 k
 libsss_nss_idmap                    x86_64                    2.6.0-1.fc35                    updates-testing                     41 k
 sssd                                x86_64                    2.6.0-1.fc35                    updates-testing                     25 k
 sssd-ad                             x86_64                    2.6.0-1.fc35                    updates-testing                    203 k
 sssd-client                         x86_64                    2.6.0-1.fc35                    updates-testing                    141 k
 sssd-common                         x86_64                    2.6.0-1.fc35                    updates-testing                    1.5 M
 sssd-common-pac                     x86_64                    2.6.0-1.fc35                    updates-testing                     89 k
 sssd-ipa                            x86_64                    2.6.0-1.fc35                    updates-testing                    269 k
 sssd-kcm                            x86_64                    2.6.0-1.fc35                    updates-testing                    104 k
 sssd-krb5                           x86_64                    2.6.0-1.fc35                    updates-testing                     76 k
 sssd-krb5-common                    x86_64                    2.6.0-1.fc35                    updates-testing                     82 k
 sssd-ldap                           x86_64                    2.6.0-1.fc35                    updates-testing                    153 k
 sssd-nfs-idmap                      x86_64                    2.6.0-1.fc35                    updates-testing                     36 k
 sssd-proxy                          x86_64                    2.6.0-1.fc35                    updates-testing                     66 k

Transaction Summary
========================================================================================================================================
Upgrade  16 Packages

Total download size: 2.9 M
Is this ok [y/N]: y
Downloading Packages:
(1/16): libipa_hbac-2.6.0-1.fc35.x86_64.rpm                                                             355 kB/s |  33 kB     00:00
(2/16): libsss_idmap-2.6.0-1.fc35.x86_64.rpm                                                            400 kB/s |  39 kB     00:00
(3/16): libsss_nss_idmap-2.6.0-1.fc35.x86_64.rpm                                                        1.0 MB/s |  41 kB     00:00
(4/16): libsss_certmap-2.6.0-1.fc35.x86_64.rpm                                                          550 kB/s |  76 kB     00:00
(5/16): sssd-2.6.0-1.fc35.x86_64.rpm                                                                    568 kB/s |  25 kB     00:00
(6/16): sssd-client-2.6.0-1.fc35.x86_64.rpm                                                             1.5 MB/s | 141 kB     00:00
(7/16): sssd-ad-2.6.0-1.fc35.x86_64.rpm                                                                 1.7 MB/s | 203 kB     00:00
(8/16): sssd-common-pac-2.6.0-1.fc35.x86_64.rpm                                                         1.9 MB/s |  89 kB     00:00
(9/16): sssd-kcm-2.6.0-1.fc35.x86_64.rpm                                                                1.2 MB/s | 104 kB     00:00
(10/16): sssd-ipa-2.6.0-1.fc35.x86_64.rpm                                                               1.6 MB/s | 269 kB     00:00
(11/16): sssd-krb5-2.6.0-1.fc35.x86_64.rpm                                                              1.3 MB/s |  76 kB     00:00
(12/16): sssd-krb5-common-2.6.0-1.fc35.x86_64.rpm                                                       1.6 MB/s |  82 kB     00:00
(13/16): sssd-nfs-idmap-2.6.0-1.fc35.x86_64.rpm                                                         1.0 MB/s |  36 kB     00:00
(14/16): sssd-ldap-2.6.0-1.fc35.x86_64.rpm                                                              1.2 MB/s | 153 kB     00:00
(15/16): sssd-proxy-2.6.0-1.fc35.x86_64.rpm                                                             1.4 MB/s |  66 kB     00:00
(16/16): sssd-common-2.6.0-1.fc35.x86_64.rpm                                                            1.4 MB/s | 1.5 MB     00:01
----------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                   1.8 MB/s | 2.9 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                1/1
  Upgrading        : libsss_idmap-2.6.0-1.fc35.x86_64                                                                              1/32
  Upgrading        : libsss_certmap-2.6.0-1.fc35.x86_64                                                                            2/32
  Upgrading        : sssd-nfs-idmap-2.6.0-1.fc35.x86_64                                                                            3/32
  Upgrading        : libsss_nss_idmap-2.6.0-1.fc35.x86_64                                                                          4/32
  Upgrading        : sssd-client-2.6.0-1.fc35.x86_64                                                                               5/32
  Running scriptlet: sssd-client-2.6.0-1.fc35.x86_64                                                                               5/32
  Upgrading        : sssd-common-2.6.0-1.fc35.x86_64                                                                               6/32
  Running scriptlet: sssd-common-2.6.0-1.fc35.x86_64                                                                               6/32
  Upgrading        : sssd-krb5-common-2.6.0-1.fc35.x86_64                                                                          7/32
  Upgrading        : sssd-common-pac-2.6.0-1.fc35.x86_64                                                                           8/32
  Upgrading        : sssd-ad-2.6.0-1.fc35.x86_64                                                                                   9/32
  Upgrading        : sssd-krb5-2.6.0-1.fc35.x86_64                                                                                10/32
  Upgrading        : sssd-ldap-2.6.0-1.fc35.x86_64                                                                                11/32
  Upgrading        : sssd-proxy-2.6.0-1.fc35.x86_64                                                                               12/32
  Upgrading        : libipa_hbac-2.6.0-1.fc35.x86_64                                                                              13/32
  Upgrading        : sssd-ipa-2.6.0-1.fc35.x86_64                                                                                 14/32
  Upgrading        : sssd-2.6.0-1.fc35.x86_64                                                                                     15/32
  Upgrading        : sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 16/32
  Running scriptlet: sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 16/32
  Cleanup          : sssd-2.5.2-5.fc35.x86_64                                                                                     17/32
  Cleanup          : sssd-ipa-2.5.2-5.fc35.x86_64                                                                                 18/32
  Cleanup          : sssd-ad-2.5.2-5.fc35.x86_64                                                                                  19/32
  Cleanup          : sssd-ldap-2.5.2-5.fc35.x86_64                                                                                20/32
  Cleanup          : sssd-common-pac-2.5.2-5.fc35.x86_64                                                                          21/32
  Cleanup          : sssd-krb5-2.5.2-5.fc35.x86_64                                                                                22/32
  Cleanup          : sssd-proxy-2.5.2-5.fc35.x86_64                                                                               23/32
  Running scriptlet: sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 24/32
  Cleanup          : sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 24/32
  Running scriptlet: sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 24/32
  Cleanup          : sssd-krb5-common-2.5.2-5.fc35.x86_64                                                                         25/32
  Running scriptlet: sssd-common-2.5.2-5.fc35.x86_64                                                                              26/32
  Cleanup          : sssd-common-2.5.2-5.fc35.x86_64                                                                              26/32
  Running scriptlet: sssd-common-2.5.2-5.fc35.x86_64                                                                              26/32
  Running scriptlet: sssd-client-2.5.2-5.fc35.x86_64                                                                              27/32
  Cleanup          : sssd-client-2.5.2-5.fc35.x86_64                                                                              27/32
  Cleanup          : libsss_idmap-2.5.2-5.fc35.x86_64                                                                             28/32
  Cleanup          : libsss_nss_idmap-2.5.2-5.fc35.x86_64                                                                         29/32
  Cleanup          : libsss_certmap-2.5.2-5.fc35.x86_64                                                                           30/32
  Cleanup          : sssd-nfs-idmap-2.5.2-5.fc35.x86_64                                                                           31/32
  Cleanup          : libipa_hbac-2.5.2-5.fc35.x86_64                                                                              32/32
  Running scriptlet: sssd-common-2.6.0-1.fc35.x86_64                                                                              32/32
  Running scriptlet: libipa_hbac-2.5.2-5.fc35.x86_64                                                                              32/32
  Verifying        : libipa_hbac-2.6.0-1.fc35.x86_64                                                                               1/32
  Verifying        : libipa_hbac-2.5.2-5.fc35.x86_64                                                                               2/32
  Verifying        : libsss_certmap-2.6.0-1.fc35.x86_64                                                                            3/32
  Verifying        : libsss_certmap-2.5.2-5.fc35.x86_64                                                                            4/32
  Verifying        : libsss_idmap-2.6.0-1.fc35.x86_64                                                                              5/32
  Verifying        : libsss_idmap-2.5.2-5.fc35.x86_64                                                                              6/32
  Verifying        : libsss_nss_idmap-2.6.0-1.fc35.x86_64                                                                          7/32
  Verifying        : libsss_nss_idmap-2.5.2-5.fc35.x86_64                                                                          8/32
  Verifying        : sssd-2.6.0-1.fc35.x86_64                                                                                      9/32
  Verifying        : sssd-2.5.2-5.fc35.x86_64                                                                                     10/32
  Verifying        : sssd-ad-2.6.0-1.fc35.x86_64                                                                                  11/32
  Verifying        : sssd-ad-2.5.2-5.fc35.x86_64                                                                                  12/32
  Verifying        : sssd-client-2.6.0-1.fc35.x86_64                                                                              13/32
  Verifying        : sssd-client-2.5.2-5.fc35.x86_64                                                                              14/32
  Verifying        : sssd-common-2.6.0-1.fc35.x86_64                                                                              15/32
  Verifying        : sssd-common-2.5.2-5.fc35.x86_64                                                                              16/32
  Verifying        : sssd-common-pac-2.6.0-1.fc35.x86_64                                                                          17/32
  Verifying        : sssd-common-pac-2.5.2-5.fc35.x86_64                                                                          18/32
  Verifying        : sssd-ipa-2.6.0-1.fc35.x86_64                                                                                 19/32
  Verifying        : sssd-ipa-2.5.2-5.fc35.x86_64                                                                                 20/32
  Verifying        : sssd-kcm-2.6.0-1.fc35.x86_64                                                                                 21/32
  Verifying        : sssd-kcm-2.5.2-5.fc35.x86_64                                                                                 22/32
  Verifying        : sssd-krb5-2.6.0-1.fc35.x86_64                                                                                23/32
  Verifying        : sssd-krb5-2.5.2-5.fc35.x86_64                                                                                24/32
  Verifying        : sssd-krb5-common-2.6.0-1.fc35.x86_64                                                                         25/32
  Verifying        : sssd-krb5-common-2.5.2-5.fc35.x86_64                                                                         26/32
  Verifying        : sssd-ldap-2.6.0-1.fc35.x86_64                                                                                27/32
  Verifying        : sssd-ldap-2.5.2-5.fc35.x86_64                                                                                28/32
  Verifying        : sssd-nfs-idmap-2.6.0-1.fc35.x86_64                                                                           29/32
  Verifying        : sssd-nfs-idmap-2.5.2-5.fc35.x86_64                                                                           30/32
  Verifying        : sssd-proxy-2.6.0-1.fc35.x86_64                                                                               31/32
  Verifying        : sssd-proxy-2.5.2-5.fc35.x86_64                                                                               32/32
You should restart:
  <snipped>

For more information run:
    sudo tracer -iat 1635175380.9097939

Upgraded:
  libipa_hbac-2.6.0-1.fc35.x86_64               libsss_certmap-2.6.0-1.fc35.x86_64          libsss_idmap-2.6.0-1.fc35.x86_64
  libsss_nss_idmap-2.6.0-1.fc35.x86_64          sssd-2.6.0-1.fc35.x86_64                    sssd-ad-2.6.0-1.fc35.x86_64
  sssd-client-2.6.0-1.fc35.x86_64               sssd-common-2.6.0-1.fc35.x86_64             sssd-common-pac-2.6.0-1.fc35.x86_64
  sssd-ipa-2.6.0-1.fc35.x86_64                  sssd-kcm-2.6.0-1.fc35.x86_64                sssd-krb5-2.6.0-1.fc35.x86_64
  sssd-krb5-common-2.6.0-1.fc35.x86_64          sssd-ldap-2.6.0-1.fc35.x86_64               sssd-nfs-idmap-2.6.0-1.fc35.x86_64
  sssd-proxy-2.6.0-1.fc35.x86_64

Complete!

# enable kcm logging
(ins)[asinha@ankur  ~]$ sudo vi /etc/sssd/sssd.conf

# check permissions
(ins)[asinha@ankur  ~]$ sudo ls -lash /etc/sssd/sssd.conf
4.0K -rw-r--r--. 1 root root 23 Oct 25 16:25 /etc/sssd/sssd.conf

# restart services
(ins)[asinha@ankur  ~]$ sudo systemctl restart sssd\*

# check services
(ins)[asinha@ankur  ~]$ sudo systemctl status sssd\*
● sssd-kcm.service - SSSD Kerberos Cache Manager
     Loaded: loaded (/usr/lib/systemd/system/sssd-kcm.service; indirect; vendor preset: disabled)
     Active: active (running) since Mon 2021-10-25 16:25:26 BST; 6s ago
TriggeredBy: ● sssd-kcm.socket
       Docs: man:sssd-kcm(5)
    Process: 3707345 ExecStartPre=/usr/sbin/sssd --genconf-section=kcm (code=exited, status=4)
   Main PID: 3707346 (sssd_kcm)
      Tasks: 1 (limit: 38003)
     Memory: 1.2M
        CPU: 16ms
     CGroup: /system.slice/sssd-kcm.service
             └─3707346 /usr/libexec/sssd/sssd_kcm --uid 0 --gid 0 --logger=files

Oct 25 16:25:26 ankur.workstation systemd[1]: Starting SSSD Kerberos Cache Manager...
Oct 25 16:25:26 ankur.workstation sssd[3707345]: [sssd] [sss_ini_read_sssd_conf] (0x0020): Permission check on config file failed.
Oct 25 16:25:26 ankur.workstation sssd[3707345]: [sssd] [confdb_init_db] (0x0020): Cannot convert INI to LDIF [1432158318]: [File owner>
Oct 25 16:25:26 ankur.workstation sssd[3707345]: [sssd] [confdb_setup] (0x0010): ConfDB initialization has failed [1432158318]: File ow>
Oct 25 16:25:26 ankur.workstation sssd[3707345]: [sssd] [load_configuration] (0x0010): Unable to setup ConfDB [1432158318]: File owners>
Oct 25 16:25:26 ankur.workstation sssd[3707345]: [sssd] [main] (0x0010): SSSD couldn't load the configuration database.
Oct 25 16:25:26 ankur.workstation sssd[3707345]: SSSD couldn't load the configuration database [1432158318]: Unknown error 1432158318.
Oct 25 16:25:26 ankur.workstation systemd[1]: Started SSSD Kerberos Cache Manager.
Oct 25 16:25:26 ankur.workstation sssd_kcm[3707346]: Starting up

● sssd-kcm.socket - SSSD Kerberos Cache Manager responder socket
     Loaded: loaded (/usr/lib/systemd/system/sssd-kcm.socket; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-10-25 16:25:26 BST; 6s ago
   Triggers: ● sssd-kcm.service
       Docs: man:sssd-kcm(8)
     Listen: /run/.heim_org.h5l.kcm-socket (Stream)
        CPU: 0
     CGroup: /system.slice/sssd-kcm.socket

Oct 25 16:25:26 ankur.workstation systemd[1]: Listening on SSSD Kerberos Cache Manager responder socket.

# restart again (just to be sure)
(ins)[asinha@ankur  ~]$ sudo systemctl restart sssd-kcm.s*

# check services again
(ins)[asinha@ankur  ~]$ sudo systemctl status sssd-kcm.s*
● sssd-kcm.service - SSSD Kerberos Cache Manager
● sssd-kcm.service - SSSD Kerberos Cache Manager
     Loaded: loaded (/usr/lib/systemd/system/sssd-kcm.service; indirect; vendor preset: disabled)
     Active: active (running) since Mon 2021-10-25 16:26:01 BST; 3s ago
TriggeredBy: ● sssd-kcm.socket
       Docs: man:sssd-kcm(5)
    Process: 3708783 ExecStartPre=/usr/sbin/sssd --genconf-section=kcm (code=exited, status=4)
   Main PID: 3708784 (sssd_kcm)
      Tasks: 1 (limit: 38003)
     Memory: 1.2M
        CPU: 22ms
     CGroup: /system.slice/sssd-kcm.service
             └─3708784 /usr/libexec/sssd/sssd_kcm --uid 0 --gid 0 --logger=files

Oct 25 16:26:01 ankur.workstation systemd[1]: Starting SSSD Kerberos Cache Manager...
Oct 25 16:26:01 ankur.workstation sssd[3708783]: [sssd] [sss_ini_read_sssd_conf] (0x0020): Permission check on config file failed.
Oct 25 16:26:01 ankur.workstation sssd[3708783]: [sssd] [confdb_init_db] (0x0020): Cannot convert INI to LDIF [1432158318]: [File owner>
Oct 25 16:26:01 ankur.workstation sssd[3708783]: [sssd] [confdb_setup] (0x0010): ConfDB initialization has failed [1432158318]: File ow>
Oct 25 16:26:01 ankur.workstation sssd[3708783]: [sssd] [load_configuration] (0x0010): Unable to setup ConfDB [1432158318]: File owners>
Oct 25 16:26:01 ankur.workstation sssd[3708783]: [sssd] [main] (0x0010): SSSD couldn't load the configuration database.
Oct 25 16:26:01 ankur.workstation sssd[3708783]: SSSD couldn't load the configuration database [1432158318]: Unknown error 1432158318.
Oct 25 16:26:01 ankur.workstation systemd[1]: Started SSSD Kerberos Cache Manager.
Oct 25 16:26:01 ankur.workstation sssd_kcm[3708784]: Starting up

● sssd-kcm.socket - SSSD Kerberos Cache Manager responder socket
     Loaded: loaded (/usr/lib/systemd/system/sssd-kcm.socket; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-10-25 16:26:01 BST; 3s ago
   Triggers: ● sssd-kcm.service
       Docs: man:sssd-kcm(8)
     Listen: /run/.heim_org.h5l.kcm-socket (Stream)
        CPU: 0
     CGroup: /system.slice/sssd-kcm.socket

# test
(ins)[asinha@ankur  ~]$ klist
klist: Credentials cache I/O operation failed while resolving ccache

# with date for logs
(ins)[asinha@ankur  ~]$ date
Mon 25 Oct 16:30:39 BST 2021
(ins)[asinha@ankur  ~]$ klist
klist: Credentials cache I/O operation failed while resolving ccache
(ins)[asinha@ankur  ~]$ kinit ankursinha
kinit: Credentials cache I/O operation failed while getting default ccache
(ins)[asinha@ankur  ~]$ 

Log attached

Comment 15 Ankur Sinha (FranciscoD) 2021-10-25 15:38:13 UTC
For this second machine, I've also saved secrets.ldb. I can confirm that removing it from /var/lib/sss/secrets and restarting the services `systemctl restart sssd\*` makes kinit etc. work again:

(ins)[asinha@ankur  ~]$ sudo mv /var/lib/sss/secrets/secrets.ldb .
(ins)[asinha@ankur  ~]$ klist
klist: Credentials cache I/O operation failed while resolving ccache
(ins)[asinha@ankur  ~]$ sudo ls /var/lib/sss/secrets/
(ins)[asinha@ankur  ~]$ date
Mon 25 Oct 16:35:13 BST 2021
(ins)[asinha@ankur  ~]$ kinit ankursinha
kinit: Credentials cache I/O operation failed while getting default ccache
(ins)[asinha@ankur  ~]$ sudo systemctl restart sssd-kcm.s*
(ins)[asinha@ankur  ~]$ kinit ankursinha
Password for ankursinha: 
(ins)[asinha@ankur  ~]$ klist
Ticket cache: KCM:1000
Default principal: ankursinha

Valid starting     Expires            Service principal
25/10/21 16:35:35  26/10/21 16:35:27  krbtgt/FEDORAPROJECT.ORG
        renew until 01/11/21 15:35:27


I've left the first machine as is so I can help test any fixes, also kept a copy of secrets.ldb.

Thanks very much for all your help.

Comment 16 Alexey Tikhonov 2021-10-25 19:25:18 UTC
(In reply to Ankur Sinha (FranciscoD) from comment #13)
> 
> > 
> > I think it should, but I'd like to ask you to keep you ccache intact until I
> > was able to reproduce this locally.
> > 
> 
> sssd_kcm.log attached again.

Thanks a lot, I was able to reproduce this locally.

What I missed while testing https://github.com/SSSD/sssd/pull/5766 is using `kswitch` (with credentials in old format) that creates 'dn=cn=default,cn=$uid,cn=persistent,cn=kcm' entry in secrets db.

Comment 17 Alexey Tikhonov 2021-10-26 21:22:48 UTC
Upstream PR: https://github.com/SSSD/sssd/pull/5841

Scratch build for testing purpose for F35: https://koji.fedoraproject.org/koji/taskinfo?taskID=77873466
Ankur, if feasible, testing welcome.

First operation like `klist` should purge entries in old format from "/var/lib/sss/secrets/secrets.ldb". Using my reproducer result is:
```
$ klist
klist: Credentials cache 'KCM:1000' not found
```

Comment 18 Ankur Sinha (FranciscoD) 2021-10-27 09:12:58 UTC
The build works for me (this is after a reboot, though---I forgot to check what it said immediately after the update, sorry).

It worked for me immediately after the update, even before I'd restarted the services, but then when I tried to restart the services just to be sure, I had sssd.service fail and so I rebooted just to be sure. All good now, no issues at all.


(ins)[asinha@ankur  ~]$ klist                                                                                                     [1/13]
Ticket cache: KCM:1000                                              
Default principal: ankursinha
                                                                    
Valid starting     Expires            Service principal                                                                                 
27/10/21 10:07:58  28/10/21 10:07:58  krbtgt/FEDORAPROJECT.ORG
        renew until 03/11/21 09:07:58
(ins)[asinha@ankur  ~]$ kdestroy                                    
(ins)[asinha@ankur  ~]$ klist                                       
klist: Credentials cache 'KCM:1000' not found
(ins)[asinha@ankur  ~]$ kinit ankursinha
Password for ankursinha: 
(ins)[asinha@ankur  ~]$ klist                                       
Ticket cache: KCM:1000       
Default principal: ankursinha
                                  
Valid starting     Expires            Service principal
27/10/21 10:08:40  28/10/21 10:08:36  krbtgt/FEDORAPROJECT.ORG
        renew until 03/11/21 09:08:36

(ins)[asinha@ankur  ~]$ rpm -qa sssd\*
sssd-nfs-idmap-2.6.0-1.fc35.1.x86_64
sssd-client-2.6.0-1.fc35.1.x86_64
sssd-common-2.6.0-1.fc35.1.x86_64
sssd-krb5-common-2.6.0-1.fc35.1.x86_64
sssd-common-pac-2.6.0-1.fc35.1.x86_64
sssd-ad-2.6.0-1.fc35.1.x86_64
sssd-krb5-2.6.0-1.fc35.1.x86_64
sssd-ldap-2.6.0-1.fc35.1.x86_64
sssd-proxy-2.6.0-1.fc35.1.x86_64
sssd-ipa-2.6.0-1.fc35.1.x86_64
sssd-2.6.0-1.fc35.1.x86_64
sssd-kcm-2.6.0-1.fc35.1.x86_64

Comment 19 Alexey Tikhonov 2021-10-27 09:56:01 UTC
(In reply to Ankur Sinha (FranciscoD) from comment #18)
> The build works for me

Thank you for testing.

Comment 20 Fedora Update System 2021-11-02 13:27:43 UTC
FEDORA-2021-37b25467d1 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-37b25467d1

Comment 21 Fedora Update System 2021-11-03 01:36:21 UTC
FEDORA-2021-37b25467d1 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-37b25467d1`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-37b25467d1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 22 Fedora Update System 2021-11-05 01:08:13 UTC
FEDORA-2021-37b25467d1 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.