Bug 1748292 - systemctl status sssd says No such file or directory about "default" when keytab exists but is empty file
Summary: systemctl status sssd says No such file or directory about "default" when key...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sssd
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.2
Assignee: Alexey Tikhonov
QA Contact: sssd-qe
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-03 10:06 UTC by Jan Pazdziora
Modified: 2020-05-02 19:13 UTC (History)
11 users (show)

Fixed In Version: sssd-2.2.3-2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-28 16:56:04 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github SSSD sssd issues 5046 0 None closed systemctl status sssd says No such file or directory about "default" when keytab exists but is empty file 2021-02-09 18:36:28 UTC
Red Hat Product Errata RHBA-2020:1863 0 None None None 2020-04-28 16:56:17 UTC

Description Jan Pazdziora 2019-09-03 10:06:37 UTC
Description of problem:

When the /etc/krb5.keytab file exists but is empty for whatever reason, the error message given by SSSD in journal is confusing, citing missing file "default" as a reason.

Version-Release number of selected component (if applicable):

sssd-krb5-common-2.2.0-16.el8.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have RHEL 8 and IPA-enrol it using ipa-client-install.
2. Check that /etc/krb5.keytab exists and is non-empty.
3. Run echo -n > /etc/krb5.keytab
4. Run systemctl restart sssd
5. Run systemctl status sssd

Actual results:

Job for sssd.service failed because the control process exited with error code.
See "systemctl status sssd.service" and "journalctl -xe" for details.

● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-09-03 06:03:24 EDT; 21s ago
  Process: 16431 ExecStart=/usr/sbin/sssd -i ${DEBUG_LOGGER} (code=exited, status=1/FAILURE)
 Main PID: 16431 (code=exited, status=1/FAILURE)

Sep 03 06:03:23 machine.example.com sssd[sudo][16443]: Starting up
Sep 03 06:03:23 machine.example.com sssd[pam][16444]: Starting up
Sep 03 06:03:23 machine.example.com sssd[pac][16445]: Starting up
Sep 03 06:03:24 machine.example.com sssd[be[example.test]][16446]: Starting up
Sep 03 06:03:24 machine.example.com sssd[be[example.test]][16446]: Failed to read keytab [default]: No such file or directory
Sep 03 06:03:24 machine.example.com sssd[16431]: Exiting the SSSD. Could not restart critical service [example.test].
Sep 03 06:03:24 machine.example.com sssd[be[implicit_files]][16432]: Shutting down
Sep 03 06:03:24 machine.example.com systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
Sep 03 06:03:24 machine.example.com systemd[1]: sssd.service: Failed with result 'exit-code'.
Sep 03 06:03:24 machine.example.com systemd[1]: Failed to start System Security Services Daemon.

Expected results:

I'd expect the "Failed to read keytab" error message to read something like

Sep 03 06:03:24 machine.example.com sssd[be[example.test]][16446]: Failed to read keytab [/etc/krb5.keytab]: It exists but does not look like a valid keytab file

Additional info:

Comment 1 Lukas Slebodnik 2019-09-03 10:49:48 UTC
Or similar error as klist.

sh$ echo -n > ./krb5.keytab
sh$ klist -kt ./krb5.keytab
Keytab name: FILE:./krb5.keytab
klist: Unsupported key table format version number while starting keytab scan

Comment 2 Alexey Tikhonov 2019-09-12 17:17:05 UTC
Upstream ticket:
https://pagure.io/SSSD/sssd/issue/4081

Comment 3 Alexey Tikhonov 2019-09-13 10:24:45 UTC
Upstream PR: https://github.com/SSSD/sssd/pull/883

With this patch in described case:

 - journal says:
```
Failed to read keytab [default]: No suitable principal found in keytab
```

 - /var/log/sssd_$backend.log says:
```
[find_principal_in_keytab] (0x0020): krb5_kt_start_seq_get failed: Unsupported key table format version number
```
(getting exactly this message into journal log is a little bit tricky)


Jan, is this ok from your point of view?

Comment 4 Jan Pazdziora 2019-09-13 10:56:04 UTC
Why would it say "default" and not "/etc/krb5.keytab"?

Comment 5 Alexey Tikhonov 2019-09-13 12:46:37 UTC
(In reply to Jan Pazdziora from comment #4)
> Why would it say "default" and not "/etc/krb5.keytab"?

There are options 'krb5_keytab' and 'ldap_krb5_keytab' that can be set in sssd config to specify location of the keytab.

If those options are not set sssd assumes "default" keytab location and uses `krb5_kt_default` to resolve default key table.
Formally speaking sssd doesn't "know" what is keytab location in this case thus prints "default".

Comment 6 Alexey Tikhonov 2019-10-17 14:30:03 UTC
(In reply to Alexey Tikhonov from comment #3)
> Upstream PR: https://github.com/SSSD/sssd/pull/883

PR was updated to include libkrb5 error in journal and to print full path to keytab in case default is used

Comment 7 Alexey Tikhonov 2019-10-21 10:14:38 UTC
    master
        e778fa1 - util/sss_krb5: debug messages fixes
        716aeba - util/sss_krb5: fixed few memory handling issues
        8f27546 - util/sss_krb5: find_principal_in_keytab() was amended
        5086353 - util/sss_krb5.c: elimination of unreachable code

Commit e778fa1 fixes this issue

Comment 8 Michal Zidek 2020-01-15 12:12:54 UTC
This bug was fixed as part of the rebase we did in RHEL 8.2.0. It would be good to fully ack it and include in the erratum.

Comment 16 Niranjan Mallapadi Raghavender 2020-03-03 10:55:21 UTC
versions:
-----------
sssd-client-2.2.3-18.el8.x86_64
sssd-ipa-2.2.3-18.el8.x86_64
sssd-kcm-2.2.3-18.el8.x86_64
sssd-dbus-2.2.3-18.el8.x86_64
sssd-2.2.3-18.el8.x86_64
sssd-nfs-idmap-2.2.3-16.el8.x86_64
python3-sssdconfig-2.2.3-18.el8.noarch
sssd-common-pac-2.2.3-18.el8.x86_64
sssd-ldap-2.2.3-18.el8.x86_64
sssd-tools-2.2.3-18.el8.x86_64
sssd-common-2.2.3-18.el8.x86_64
sssd-ad-2.2.3-18.el8.x86_64
sssd-krb5-common-2.2.3-18.el8.x86_64
sssd-proxy-2.2.3-18.el8.x86_64
sssd-winbind-idmap-2.2.3-18.el8.x86_64
sssd-krb5-2.2.3-18.el8.x86_64


# Join system to IPA using ipa client 

# move /etc/krb5.keytab 
mv /etc/krb5.keytab /etc/krb5.keytab.orig

# create an empty keytab file
echo -n > /etc/krb5.keytab

# start sssd 

<snip>
Mar 03 16:16:44 client1.testrealm.test sssd[25801]: Starting up
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: Starting up
Mar 03 16:16:44 client1.testrealm.test sssd[be[implicit_files]][25802]: Starting up
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25803]: Failed to read keytab [FILE:/etc/krb5.keytab]: No suitable principal found in keytab
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: Starting up
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:16:44 client1.testrealm.test sssd[be[testrealm.test]][25804]: krb5_kt_start_seq_get failed: Unsupported key table format version number
</snip>


# Join system to AD 

[root@client1 ~]# realm join -U Administrator -v SARABHAI.TEST
 * Resolving: _ldap._tcp.sarabhai.test
 * Performing LDAP DSE lookup on: 192.168.122.216
 * Successfully discovered: SARABHAI.TEST
Password for Administrator:
 * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/sbin/adcli
 * LANG=C /usr/sbin/adcli join --verbose --domain SARABHAI.TEST --domain-realm SARABHAI.TEST --domain-controller 192.168.122.216 --login-type user --login-user Administrator --stdin-password
 * Using domain name: SARABHAI.TEST
 * Calculated computer account name from fqdn: CLIENT1
 * Using domain realm: SARABHAI.TEST
 * Sending netlogon pings to domain controller: cldap://192.168.122.216
 * Received NetLogon info from: vikram.SARABHAI.TEST
 * Wrote out krb5.conf snippet to /var/cache/realmd/adcli-krb5-2lS4g4/krb5.d/adcli-krb5-conf-OJyy4d
 * Authenticated as user: Administrator@SARABHAI.TEST
 * Using GSS-SPNEGO for SASL bind
 * Looked up short domain name: SARABHAI
 * Looked up domain SID: S-1-5-21-1672089527-2408710569-2399489135
 * Using fully qualified name: client1.testrealm.test
 * Using domain name: SARABHAI.TEST
 * Using computer account name: CLIENT1
 * Using domain realm: SARABHAI.TEST
 * Calculated computer account name from fqdn: CLIENT1
 * Generated 120 character computer password
 * Using keytab: FILE:/etc/krb5.keytab
 * Computer account for CLIENT1$ does not exist
 * Found well known computer container at: CN=Computers,DC=SARABHAI,DC=TEST
 * Calculated computer account: CN=CLIENT1,CN=Computers,DC=SARABHAI,DC=TEST
 * Created computer account: CN=CLIENT1,CN=Computers,DC=SARABHAI,DC=TEST
 * Sending netlogon pings to domain controller: cldap://192.168.122.216
 * Received NetLogon info from: vikram.SARABHAI.TEST
 * Set computer password
 * Retrieved kvno '2' for computer account in directory: CN=CLIENT1,CN=Computers,DC=SARABHAI,DC=TEST
 * Checking RestrictedKrbHost/client1.testrealm.test
 *    Added RestrictedKrbHost/client1.testrealm.test
 * Checking RestrictedKrbHost/CLIENT1
 *    Added RestrictedKrbHost/CLIENT1
 * Checking host/client1.testrealm.test
 *    Added host/client1.testrealm.test
 * Checking host/CLIENT1
 *    Added host/CLIENT1
 * Discovered which keytab salt to use
 * Added the entries to the keytab: CLIENT1$@SARABHAI.TEST: FILE:/etc/krb5.keytab
 * Added the entries to the keytab: host/CLIENT1@SARABHAI.TEST: FILE:/etc/krb5.keytab
 * Added the entries to the keytab: host/client1.testrealm.test@SARABHAI.TEST: FILE:/etc/krb5.keytab
 * Added the entries to the keytab: RestrictedKrbHost/CLIENT1@SARABHAI.TEST: FILE:/etc/krb5.keytab
 * Added the entries to the keytab: RestrictedKrbHost/client1.testrealm.test@SARABHAI.TEST: FILE:/etc/krb5.keytab
 * /usr/bin/systemctl enable sssd.service
Created symlink /etc/systemd/system/multi-user.target.wants/sssd.service → /usr/lib/systemd/system/sssd.service.
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/bin/authselect select sssd with-mkhomedir --force && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service
Profile "sssd" was selected.
The following nsswitch maps are overwritten by the profile:
- passwd
- group
- netgroup
- automount
- automount
- services

Make sure that SSSD service is configured and enabled. See SSSD documentation for more information.

- with-mkhomedir is selected, make sure pam_oddjob_mkhomedir module
  is present and oddjobd service is enabled
  - systemctl enable oddjobd.service
  - systemctl start oddjobd.service

Created symlink /etc/systemd/system/multi-user.target.wants/oddjobd.service → /usr/lib/systemd/system/oddjobd.service.
 * Successfully enrolled machine in realm

[root@client1 ~]# cat /etc/sssd/sssd.conf

[sssd]
domains = SARABHAI.TEST
config_file_version = 2
services = nss, pam

[domain/SARABHAI.TEST]
ad_domain = SARABHAI.TEST
krb5_realm = SARABHAI.TEST
realmd_tags = manages-system joined-with-adcli 
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad

[root@client1 ~]# systemctl restart sssd
Job for sssd.service failed because the control process exited with error code.
See "systemctl status sssd.service" and "journalctl -xe" for details.
[root@client1 ~]# journalctl -xe
Mar 03 16:24:46 client1.testrealm.test sssd[be[SARABHAI.TEST]][27246]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found
Mar 03 16:24:46 client1.testrealm.test sssd[be[SARABHAI.TEST]][27246]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found
Mar 03 16:24:46 client1.testrealm.test sssd[be[SARABHAI.TEST]][27246]: Failed to read keytab [FILE:/etc/krb5.keytab]: No suitable principal found in keytab
Mar 03 16:24:46 client1.testrealm.test sssd[27220]: Exiting the SSSD. Could not restart critical service [SARABHAI.TEST].
Mar 03 16:24:46 client1.testrealm.test sssd[be[implicit_files]][27233]: Shutting down (status = 0)
Mar 03 16:24:46 client1.testrealm.test systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
Mar 03 16:24:46 client1.testrealm.test systemd[1]: sssd.service: Failed with result 'exit-code'.
Mar 03 16:24:46 client1.testrealm.test systemd[1]: Failed to start System Security Services Daemon.


Marking it as verified based on above results.

Comment 17 Niranjan Mallapadi Raghavender 2020-03-03 11:00:45 UTC
In the previous comment AD was tested with no keytab. Retesting with blank keytab

[root@client1 ~]# mv /etc/krb5.keytab /etc/krb5.keytab.backup
[root@client1 ~]# echo -n > /etc/krb5.keytab
[root@client1 ~]# systemctl restart sssd
Job for sssd.service failed because the control process exited with error code.
See "systemctl status sssd.service" and "journalctl -xe" for details.
[root@client1 ~]# journalctl -xe
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sssd.service has failed.
--
-- The result is failed.  
Mar 03 16:29:35 client1.testrealm.test systemd[1]: sssd.service: Service RestartSec=100ms expired, scheduling restart.
Mar 03 16:29:35 client1.testrealm.test systemd[1]: sssd.service: Scheduled restart job, restart counter is at 1.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Automatic restarting of the unit sssd.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Mar 03 16:29:35 client1.testrealm.test systemd[1]: Stopped System Security Services Daemon.
-- Subject: Unit sssd.service has finished shutting down
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sssd.service has finished shutting down.
Mar 03 16:29:35 client1.testrealm.test systemd[1]: Starting System Security Services Daemon...
-- Subject: Unit sssd.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sssd.service has begun starting up.
Mar 03 16:29:35 client1.testrealm.test sssd[27870]: Starting up
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: Starting up
Mar 03 16:29:35 client1.testrealm.test sssd[be[implicit_files]][27871]: Starting up
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: krb5_kt_start_seq_get failed: Unsupported key table format version number
Mar 03 16:29:35 client1.testrealm.test sssd[be[SARABHAI.TEST]][27872]: Failed to read keytab [FILE:/etc/krb5.keytab]: No suitable principal found in keytab
Mar 03 16:29:36 client1.testrealm.test sssd[be[SARABHAI.TEST]][27873]: Starting up

Comment 19 errata-xmlrpc 2020-04-28 16:56:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1863


Note You need to log in before you can comment on or make changes to this bug.