Bug 2212343

Summary: kinit: Connection refused while getting default ccache
Product: [Fedora] Fedora Reporter: Tomas Hofman <thofman>
Component: sssdAssignee: sssd-maintainers <sssd-maintainers>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 38CC: abokovoy, atikhono, jhrozek, lslebodn, luk.claes, mzidek, pbrezina, sbose, ssorce, sssd-maintainers
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-06 14:22:14 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
strace if the kinit command none

Description Tomas Hofman 2023-06-05 11:05:19 UTC
Hello, on freshly installed system, after installing krb5-workstation and configuring /etc/krb5.conf, I'm unable to use `kinit` command:

```
$ kinit thofman 
kinit: Connection refused while getting default ccache
```

Complete output of `KRB5_TRACE=/dev/stderr strace -f klist` is attached bellow.

Notably, strace shows lines like:

```
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/.heim_org.h5l.kcm-socket"}, 110) = -1 ECONNREFUSED (Connection refused)
```

Not sure if this is related, but sssd is installed (was in base installation?) but does not start:

```
$ systemctl status sssd
○ sssd.service - System Security Services Daemon
     Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; preset: ena>
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
  Condition: start condition failed at Mon 2023-06-05 10:03:58 CEST; 43min ago
             ├─ ConditionPathExists=|/etc/sssd/sssd.conf was not met
             └─ ConditionDirectoryNotEmpty=|/etc/sssd/conf.d was not met

Jun 05 10:03:58 fedora systemd[1]: sssd.service - System Security Services Daem>
```

When I modify /etc/krb5.conf.d/kcm_default_ccache and comment out the line "default_ccache_name = KCM:", kinit starts to work.

Reproducible: Always

Comment 1 Tomas Hofman 2023-06-05 11:06:50 UTC
Created attachment 1969037 [details]
strace if the kinit command

Comment 2 Alexander Bokovoy 2023-06-05 11:16:41 UTC
Do you have sssd-kcm installed?

Comment 3 Tomas Hofman 2023-06-05 13:46:49 UTC
Hello Alexander, yes, sssd-kcm is installed too.

Comment 4 Simo Sorce 2023-06-05 15:03:17 UTC
What do you get if you run this as root:

# systemctl status sssd-kcm.socket

Comment 5 Tomas Hofman 2023-06-06 07:30:18 UTC
It shows:

```
$ sudo systemctl status sssd-kcm.socket
× sssd-kcm.socket - SSSD Kerberos Cache Manager responder socket
     Loaded: loaded (/usr/lib/systemd/system/sssd-kcm.socket; enabled; preset: enabled)
     Active: failed (Result: service-start-limit-hit) since Mon 2023-06-05 10:04:14 CEST; 23h ago
   Duration: 15.767s
   Triggers: ● sssd-kcm.service
       Docs: man:sssd-kcm(8)
     Listen: /run/.heim_org.h5l.kcm-socket (Stream)

Jun 05 10:03:58 fedora systemd[1]: Listening on sssd-kcm.socket - SSSD Kerberos Cache Manager responder socket.
Jun 05 10:04:14 fedora systemd[1]: sssd-kcm.socket: Failed with result 'service-start-limit-hit'.
```

I tried to restart it but it ends up in the same state. `journalctl -u sssd-kcm.service` shows:

```
Jun 06 09:18:34 fedora systemd[1]: Starting sssd-kcm.service - SSSD Kerberos Cache Manager...
Jun 06 09:18:34 fedora systemd[1]: Started sssd-kcm.service - SSSD Kerberos Cache Manager.
Jun 06 09:18:34 fedora sssd_kcm[41417]: Starting up
Jun 06 09:18:34 fedora sssd_kcm[41417]: Failed to init Kerberos context [Permission denied]
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.
Jun 06 09:18:34 fedora systemd[1]: Starting sssd-kcm.service - SSSD Kerberos Cache Manager...
Jun 06 09:18:34 fedora systemd[1]: Started sssd-kcm.service - SSSD Kerberos Cache Manager.
Jun 06 09:18:34 fedora sssd_kcm[41422]: Starting up
Jun 06 09:18:34 fedora sssd_kcm[41422]: Failed to init Kerberos context [Permission denied]
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.
Jun 06 09:18:34 fedora systemd[1]: Starting sssd-kcm.service - SSSD Kerberos Cache Manager...
Jun 06 09:18:34 fedora systemd[1]: Started sssd-kcm.service - SSSD Kerberos Cache Manager.
Jun 06 09:18:34 fedora sssd_kcm[41425]: Starting up
Jun 06 09:18:34 fedora sssd_kcm[41425]: Failed to init Kerberos context [Permission denied]
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.
Jun 06 09:18:34 fedora systemd[1]: Starting sssd-kcm.service - SSSD Kerberos Cache Manager...
Jun 06 09:18:34 fedora systemd[1]: Started sssd-kcm.service - SSSD Kerberos Cache Manager.
Jun 06 09:18:34 fedora sssd_kcm[41428]: Starting up
Jun 06 09:18:34 fedora sssd_kcm[41428]: Failed to init Kerberos context [Permission denied]
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.
Jun 06 09:18:34 fedora systemd[1]: Starting sssd-kcm.service - SSSD Kerberos Cache Manager...
Jun 06 09:18:34 fedora systemd[1]: Started sssd-kcm.service - SSSD Kerberos Cache Manager.
Jun 06 09:18:34 fedora sssd_kcm[41431]: Starting up
Jun 06 09:18:34 fedora sssd_kcm[41431]: Failed to init Kerberos context [Permission denied]
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Start request repeated too quickly.
Jun 06 09:18:34 fedora systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.
Jun 06 09:18:34 fedora systemd[1]: Failed to start sssd-kcm.service - SSSD Kerberos Cache Manager.
```

Comment 6 Alexander Bokovoy 2023-06-06 09:35:37 UTC
Jun 06 09:18:34 fedora sssd_kcm[41417]: Failed to init Kerberos context [Permission denied]

this sounds like you have some of krb5.conf-included files inaccessible. This is a known issue with krb5 library -- if when reading /etc/krb5.conf and includes it cannot read some file, the whole initialization of the library is failing.

So check that /etc/krb5.conf and includes are globally readable. We had some customers who had files in /etc/krb5.conf.d or in /var/lib/sss/pubconf/krb5.conf.d that had been unreadable to any non-root process.

Comment 7 Lukas Slebodnik 2023-06-06 12:13:34 UTC
> this sounds like you have some of krb5.conf-included files inaccessible.

Or more likely wrong SELinux file context for /etc/krb5.conf :-)
e.g.
[root@localhost ~]# ls -lZ /etc/krb5.conf
-rw-r--r--. 1 root root system_u:object_r:user_home_t:s0 880 Nov 16  2022 /etc/krb5.conf


[root@localhost ~]# systemctl status sssd-kcm
× sssd-kcm.service - SSSD Kerberos Cache Manager
     Loaded: loaded (/usr/lib/systemd/system/sssd-kcm.service; indirect; vendor preset: disabled)
     Active: failed (Result: exit-code) since Tue 2023-06-06 14:11:20 CEST; 1min 23s ago
TriggeredBy: ● sssd-kcm.socket
       Docs: man:sssd-kcm(5)
    Process: 697372 ExecStartPre=/usr/sbin/sssd --genconf-section=kcm (code=exited, status=0/SUCCESS)
    Process: 697373 ExecStart=/usr/libexec/sssd/sssd_kcm --uid 0 --gid 0 ${DEBUG_LOGGER} (code=exited, status=3)
   Main PID: 697373 (code=exited, status=3)

Jun 06 14:11:20 localhost.localdomain systemd[1]: Starting SSSD Kerberos Cache Manager...
Jun 06 14:11:20 localhost.localdomain systemd[1]: Started SSSD Kerberos Cache Manager.
Jun 06 14:11:20 localhost.localdomain sssd_kcm[697373]: Starting up
Jun 06 14:11:20 localhost.localdomain sssd_kcm[697373]: Failed to init Kerberos context [Permission denied]
Jun 06 14:11:20 localhost.localdomain systemd[1]: sssd-kcm.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 06 14:11:20 localhost.localdomain systemd[1]: sssd-kcm.service: Failed with result 'exit-code'.



[root@localhost ~]# ausearch -m avc -ts recent
----
time->Tue Jun  6 14:11:20 2023
type=AVC msg=audit(1686053480.757:10014): avc:  denied  { read } for  pid=697373 comm="sssd_kcm" name="krb5.conf" dev="dm-2" ino=6676617 scontext=system_u:system_r:sssd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

Comment 8 Tomas Hofman 2023-06-06 14:22:14 UTC
Yes! It was wrong selinux context:

$ ls -lZ /etc/krb5.conf*
-rw-r--r--. 1 root root unconfined_u:object_r:user_home_t:s0 1006 Jun  6 16:14 /etc/krb5.conf

After correcting to

-rw-r--r--. 1 root root unconfined_u:object_r:krb5_conf_t:s0 1006 Jun  6 16:15 /etc/krb5.conf

and restarting sssd-kcm.service and sssd-kcm.socket it works.

Thanks for all the help!