[this is on a ostree system (Fedora 41 Kinoite)] From what I can tell, the sssd permissions changed recently from root:sssd to sssd:sssd, and there are subtle differences in the way the config directories and config files are owned that cause the most recent update to fail. The previous version (2.10.0) had /etc/sssd at mode 0700, owned by root:sssd. Now that the service runs as sssd:sssd it is not able to access previous configs in /etc/sssd and the daemon fails with a permissions error. There are a couple of contributing factors to this update failure: 1. the three-way merge for /etc files done by ostree does not preserve file permissions, and does not update directory permissions. When /usr/etc/sssd changed from 0700 to 0750, this was not reflected in /etc/sssd. 2. the ExecStartPre= statements in sssd.service are a bit optimistic. The chown and chmod commands don't work with the current service settings (User= and Group=). They currently fail but this is masked by the '-f' option. I was able to fix this on my local system by (1) commenting out the ExecStartPre= statements and (2) moving them to a dependent service that runs as root. I guess this could also be accomplished with PermissionsStartOnly=true. Reproducible: Always Steps to Reproduce: 1. build up a system that uses sssd (fully-populated /etc/sssd/sssd.conf) 2. stop sssd 3. chmod 0700 /etc/sssd 4. chown root:sssd /etc/sssd 5. try to start sssd Actual Results: sssd does not start, it is not able to correctly re-own the files and directories it needs. Here is the update report from the most recent ostree transaction $ rpm-ostree db diff ostree diff commit from: rollback deployment (5796a4d9fbff9b2d0e28f34070e6958e0cfb5a1b3ab702fd1ed318ade3aa91ef) ostree diff commit to: booted deployment (f07128cd02dc40fbe578714f095140f44e5986f98902783d00bbd837a0ed06d6) Upgraded: cldr-emoji-annotation 1:46.1~beta2-1.fc41 -> 1:46.1-1.fc41 cldr-emoji-annotation-dtd 1:46.1~beta2-1.fc41 -> 1:46.1-1.fc41 ibus-typing-booster 2.27.1-1.fc41 -> 2.27.2-1.fc41 intel-vpl-gpu-rt 24.3.4-1.fc41 -> 24.4.4-1.fc41 libsss_certmap 2.10.0-2.fc41 -> 2.10.1-1.fc41 libsss_idmap 2.10.0-2.fc41 -> 2.10.1-1.fc41 libsss_nss_idmap 2.10.0-2.fc41 -> 2.10.1-1.fc41 libsss_sudo 2.10.0-2.fc41 -> 2.10.1-1.fc41 libva-intel-media-driver 24.3.4-1.fc41 -> 24.4.4-1.fc41 libvpl 1:2.13.0-1.fc41 -> 1:2.14.0-1.fc41 libxml2 2.12.8-2.fc41 -> 2.12.9-1.fc41 openjpeg 2.5.3-1.fc41 -> 2.5.3-2.fc41 setup 2.15.0-5.fc41 -> 2.15.0-8.fc41 sssd-client 2.10.0-2.fc41 -> 2.10.1-1.fc41 sssd-common 2.10.0-2.fc41 -> 2.10.1-1.fc41 sssd-kcm 2.10.0-2.fc41 -> 2.10.1-1.fc41 sssd-krb5-common 2.10.0-2.fc41 -> 2.10.1-1.fc41 sssd-ldap 2.10.0-2.fc41 -> 2.10.1-1.fc41 sssd-nfs-idmap 2.10.0-2.fc41 -> 2.10.1-1.fc41
Was able to fix the issue with the follow patch: --- /usr/lib/systemd/system/sssd.service 1970-01-01 01:00:00.000000000 +0100 +++ /etc/systemd/system/sssd.service 2024-12-30 00:33:11.944722822 +0100 @@ -12,7 +12,8 @@ Environment=DEBUG_LOGGER=--logger=files EnvironmentFile=-/etc/sysconfig/sssd ExecStartPre=+-/bin/chown -f -R root:sssd /etc/sssd -ExecStartPre=+-/bin/chmod -f -R g+r /etc/sssd +ExecStartPre=+-/bin/find /etc/sssd -type d -exec /bin/chmod -f 750 {} \; +ExecStartPre=+-/bin/find /etc/sssd -type f -exec /bin/chmod -f 440 {} \; ExecStartPre=+-/bin/sh -c "/bin/chown -f sssd:sssd /var/lib/sss/db/*.ldb" ExecStartPre=+-/bin/chown -f -R sssd:sssd /var/lib/sss/gpo_cache ExecStartPre=+-/bin/sh -c "/bin/chown -f sssd:sssd /var/log/sssd/*.log"
Hi, (In reply to Carl Roth from comment #0) > > 1. the three-way merge for /etc files done by ostree does not preserve file > permissions, and does not update directory permissions. When /usr/etc/sssd > changed from 0700 to 0750, this was not reflected in /etc/sssd. This is the main reason for sssd.service chown/chmod gimmicks... > > 2. the ExecStartPre= statements in sssd.service are a bit optimistic. The > chown and chmod commands don't work with the current service settings (User= > and Group=). They currently fail but this is masked by the '-f' option. All chown/chmod are prefixed with "+": ``` If the executable path is prefixed with "+" then the process is executed with full privileges. In this mode privilege restrictions configured with User=, Group=, CapabilityBoundingSet= or the various file system namespacing options (such as PrivateDevices=, PrivateTmp=) are not applied to the invoked command line ``` (In reply to Cédric Bellegarde from comment #1) > > -ExecStartPre=+-/bin/chmod -f -R g+r /etc/sssd > +ExecStartPre=+-/bin/find /etc/sssd -type d -exec /bin/chmod -f 750 {} \; > +ExecStartPre=+-/bin/find /etc/sssd -type f -exec /bin/chmod -f 440 {} \; hm... is this about missing g+x for /etc/sssd?
https://github.com/SSSD/sssd/pull/7783
I ran into this issue while updating my laptop from Fedora 40 --> 41 Can confirm patch in https://bugzilla.redhat.com/show_bug.cgi?id=2334868#c1 is good.
I also wanted to add, my installation was x86 regular (not ostree)
(In reply to bgrech from comment #5) > I also wanted to add, my installation was x86 regular (not ostree) Do you happen to have SSSD logs?
Getting the same issue after updating to 2.10.1 on a Fedora 41 workstation. Currently can't kinit on either Fedora or Red Hat internal with the message "kinit: Connection refused while getting default ccache". After trying to fix the permissions the systemctl status logs show: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found.
(In reply to Charalampos Stratakis from comment #7) > Getting the same issue after updating to 2.10.1 on a Fedora 41 workstation. > Currently can't kinit on either Fedora or Red Hat internal with the message > "kinit: Connection refused while getting default ccache". Do you have SSSD configured or do you merely use 'sssd_kcm' as a KCM crendentials storage? Could you please provide: - '/var/log/sssd/sssd_kcm.log', - 'ls -la /etc/sssd' before "fixing permissions"? Do you have rpm update log?
(In reply to Alexey Tikhonov from comment #8) > (In reply to Charalampos Stratakis from comment #7) > > Getting the same issue after updating to 2.10.1 on a Fedora 41 workstation. > > Currently can't kinit on either Fedora or Red Hat internal with the message > > "kinit: Connection refused while getting default ccache". > > Do you have SSSD configured or do you merely use 'sssd_kcm' as a KCM > crendentials storage? > > Could you please provide: > - '/var/log/sssd/sssd_kcm.log', > - 'ls -la /etc/sssd' > before "fixing permissions"? > > Do you have rpm update log? Sure, will do an rpm downgrade/upgrade so the scriptlets are run properly. The sssd_kcm.log entries are: [sssd] [access_check_file] (0x0020): Unexpected access to '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users [sssd] [load_configuration] (0x0010): Unable to setup ConfDB [1432158324]: File ownership and permissions check failed [sssd] [main] (0x0010): KCM couldn't load the configuration [1432158324]: File ownership and permissions check failed This file is: -r--r-----. 1 root sssd ls -la /etc/sssd is: drwxr-x---. 4 root sssd 4096 Dec 10 01:00 . drwxr-xr-x. 204 root root 12288 Jan 7 17:40 .. drwxr-x---. 2 root sssd 4096 Dec 10 01:00 conf.d drwxr-x---. 2 root sssd 4096 Dec 10 01:00 pki -r--r-----. 1 root sssd 2460 Aug 9 2023 sssd.conf I'll try to find the dnf logs from the update.
(In reply to Alexey Tikhonov from comment #6) > (In reply to bgrech from comment #5) > > I also wanted to add, my installation was x86 regular (not ostree) > > Do you happen to have SSSD logs? yes sssd_kcm.log i will attach
(In reply to Alexey Tikhonov from comment #8) > (In reply to Charalampos Stratakis from comment #7) > > Getting the same issue after updating to 2.10.1 on a Fedora 41 workstation. > > Currently can't kinit on either Fedora or Red Hat internal with the message > > "kinit: Connection refused while getting default ccache". > > Do you have SSSD configured or do you merely use 'sssd_kcm' as a KCM > crendentials storage? > > Could you please provide: > - '/var/log/sssd/sssd_kcm.log', > - 'ls -la /etc/sssd' > before "fixing permissions"? > > Do you have rpm update log? Unfortunately my update logs have been rotated, I distinctively remember an error related to sssd possibly due to the service being unable to restart but I don't recall anything more.
(In reply to bgrech from comment #10) > (In reply to Alexey Tikhonov from comment #6) > > (In reply to bgrech from comment #5) > > > I also wanted to add, my installation was x86 regular (not ostree) > > > > Do you happen to have SSSD logs? > > yes sssd_kcm.log > > i will attach What is the output of `ls -la /etc/sssd/conf.d/authconfig-sssd.conf`?
(In reply to Charalampos Stratakis from comment #9) > [sssd] [access_check_file] (0x0020): Unexpected access to '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users What is the output of `ls -la /etc/sssd/conf.d/ipa.redhat.com.conf`?
And what is the origin of '/etc/sssd/conf.d/authconfig-sssd.conf'?
(In reply to Alexey Tikhonov from comment #14) > (In reply to Charalampos Stratakis from comment #9) > > [sssd] [access_check_file] (0x0020): Unexpected access to '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users > > What is the output of `ls -la /etc/sssd/conf.d/ipa.redhat.com.conf`? -r--r-----. 1 root sssd
(In reply to Charalampos Stratakis from comment #16) > (In reply to Alexey Tikhonov from comment #14) > > (In reply to Charalampos Stratakis from comment #9) > > > [sssd] [access_check_file] (0x0020): Unexpected access to '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users > > > > What is the output of `ls -la /etc/sssd/conf.d/ipa.redhat.com.conf`? > > -r--r-----. 1 root sssd And 'sssd_kcm' still says "Unexpected access to '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users"?
(In reply to Alexey Tikhonov from comment #17) > (In reply to Charalampos Stratakis from comment #16) > > (In reply to Alexey Tikhonov from comment #14) > > > (In reply to Charalampos Stratakis from comment #9) > > > > [sssd] [access_check_file] (0x0020): Unexpected access to '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users > > > > > > What is the output of `ls -la /etc/sssd/conf.d/ipa.redhat.com.conf`? > > > > -r--r-----. 1 root sssd > > And 'sssd_kcm' still says "Unexpected access to > '/etc/sssd/conf.d/ipa.redhat.com.conf' by other users"? Restarted sssd-kcm and tried to restart sssd as well to see any logs updates if at all as possibly the unexpected access could be from before from tinkering with perms. So currently my sssd-kcm logs provide: (2025-01-07 18:31:05): [kcm] [server_setup] (0x3f7c0): Starting with debug level = 0x0070 (2025-01-07 18:31:05): [kcm] [activate_unix_sockets] (0x3f7c0): Warning: socket path defined in systemd unit (/run/.heim_org.h5l.kcm-socket) and sssd.conf (/var/run/.heim_org.h5l.kcm-socket) don't match (2025-01-07 18:31:05): [kcm] [server_loop] (0x3f7c0): Entering main loop under uid=963 (euid=963) : gid=963 (egid=963) with SECBIT_KEEP_CAPS = 0 and following capabilities: (nothing) And the "systemctl status sssd": × sssd.service - System Security Services Daemon Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; preset: enabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf, 50-keep-warm.conf Active: failed (Result: exit-code) since Tue 2025-01-07 18:31:49 CET; 8min ago Invocation: 4e1b18f719f64b7a8260463a95878a97 Process: 259548 ExecStartPre=/bin/chown -f -R root:sssd /etc/sssd (code=exited, status=0/SUCCESS) Process: 259550 ExecStartPre=/bin/chmod -f -R g+r /etc/sssd (code=exited, status=0/SUCCESS) Process: 259552 ExecStartPre=/bin/sh -c /bin/chown -f sssd:sssd /var/lib/sss/db/*.ldb (code=exited, status=0/SUCCESS) Process: 259554 ExecStartPre=/bin/chown -f -R sssd:sssd /var/lib/sss/gpo_cache (code=exited, status=0/SUCCESS) Process: 259556 ExecStartPre=/bin/sh -c /bin/chown -f sssd:sssd /var/log/sssd/*.log (code=exited, status=0/SUCCESS) Process: 259558 ExecStart=/usr/sbin/sssd -i ${DEBUG_LOGGER} (code=exited, status=1/FAILURE) Main PID: 259558 (code=exited, status=1/FAILURE) Mem peak: 6.6M CPU: 174ms Jan 07 18:31:49 ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jan 07 18:31:49 ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jan 07 18:31:49 ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jan 07 18:31:49 ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jan 07 18:31:49 ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jan 07 18:31:49 ldap_child[259571]: Failed to read keytab [FILE:/etc/krb5.keytab]: No suitable principal found in keytab Jan 07 18:31:49 sssd[259558]: Exiting the SSSD. Could not restart critical service [ipa.redhat.com]. Jan 07 18:31:49 systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE Jan 07 18:31:49 systemd[1]: sssd.service: Failed with result 'exit-code'. Jan 07 18:31:49 systemd[1]: Failed to start sssd.service - System Security Services Daemon. Attaching the latest snippet of /var/log/sssd/sssd.log as well.
Noting also that I just tried to kinit and it works successfully. sssd is still down but my kinit started working for both Fedora and internal Red Hat.
(In reply to Alexey Tikhonov from comment #13) > (In reply to bgrech from comment #10) > > (In reply to Alexey Tikhonov from comment #6) > > > (In reply to bgrech from comment #5) > > > > I also wanted to add, my installation was x86 regular (not ostree) > > > > > > Do you happen to have SSSD logs? > > > > yes sssd_kcm.log > > > > i will attach > > What is the output of `ls -la /etc/sssd/conf.d/authconfig-sssd.conf`? -r--r-----. 1 root sssd 382 Jan 6 14:12 /etc/sssd/conf.d/authconfig-sssd.conf (In reply to Charalampos Stratakis from comment #20) > Noting also that I just tried to kinit and it works successfully. sssd is > still down but my kinit started working for both Fedora and internal Red Hat. I have the same behavior also, sssd-kcm however is now running though after I applied the patch mentioned above
> ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Does '/etc/krb5.keytab' exist?
(In reply to Alexey Tikhonov from comment #22) > > ldap_child[259571]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found > > Does '/etc/krb5.keytab' exist? It does not, I haven't dealt with a keytab file before though.
Could anyone please share /var/log/sssd/sssd_$domain.log and ldap_child.log (where SSSD doesn't start despite /etc/sssd/* permissions being fixed)?
So far looks like two issues are mixed in this ticket: (1) missing 'g+x' bit on /etc/sssd and sub-folders (2) additional config snippets under /etc/sssd/conf.d/ that was *not* used previously but *are used* now and containing bad configuration (so that SSSD can't start)
Pushed PR: https://github.com/SSSD/sssd/pull/7783 * `master` * 561c51bd799246563018fb76dcfe42d64b5cb348 - SYSTEMD: fix missing 'g+x' on /etc/sssd and subdirs * 4b35ac30cb7e643bd4c7eb9d2492f667af59ee4a - SYSTEMD: traverse 'sssdconfdir' symlink while chown-ing * `sssd-2-10` * a0ab704c94ebaf83aeeb51d06846d39146a64706 - SYSTEMD: fix missing 'g+x' on /etc/sssd and subdirs * fa26915e6cabcd21887e7ac0887dbffc0cd03bf2 - SYSTEMD: traverse 'sssdconfdir' symlink while chown-ing
FEDORA-2025-49004d4e5e (sssd-2.10.2-1.fc41) has been submitted as an update to Fedora 41. https://bodhi.fedoraproject.org/updates/FEDORA-2025-49004d4e5e
FEDORA-2025-49004d4e5e has been pushed to the Fedora 41 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-49004d4e5e` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-49004d4e5e See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-49004d4e5e (sssd-2.10.2-1.fc41) has been pushed to the Fedora 41 stable repository. If problem still persists, please make note of it in this bug report.