Created attachment 1870873 [details]
sssd_nss.log

(In reply to Micah Abbott from comment #0)
> 
> https://github.com/SSSD/sssd/pull/6075
> 
> ...may resolve this issue for us.

This may "hide" issue, but not resolve it.

Do you have a custom sssd.conf? Could you please share content of /etc/sssd/* and also /var/log/sssd.log?

This problem is being reproduced in an ephemeral CI cluster, so access to the nodes is difficult as they are torn down after failure.

However, looking at a single RHCOS node (outside of the cluster), the contents of `/etc/sssd/`:


```
$ sudo ls -lR /etc/sssd/
/etc/sssd/:
total 0
drwx--x--x. 2 sssd sssd 6 Apr  5 14:31 conf.d
drwx--x--x. 2 root root 6 Apr  5 14:31 pki

/etc/sssd/conf.d:
total 0

/etc/sssd/pki:
total 0
```

> [sss_user_by_name_or_uid] (0x0040): [sssd] is neither a valid UID nor a user name which could be resolved by getpwnam()

This happens in `nss_process_init()`->`sssd_supplementary_group()`->`sss_user_by_name_or_uid(SSSD_USER)`:
https://github.com/SSSD/sssd/blob/d1bce130f590e7e81a8472b8c9804ebe63898852/src/responder/nss/nsssrv.c#L404

For RHEL8 SSSD is configured `--with-sssd-user=sssd`

`%pre ipa, krb5-common, common, proxy` sections of a RHEL8 spec-file create this local user.
But it seems this user is missing on your host. Could you please confirm this?

What package is used by RHCOS? Is it the same as for RHEL? How is it installed?

That would explain it.

```
$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
core:x:1000:1000:CoreOS Admin:/var/home/core:/bin/bash
containers:x:1001:995:User for housing the sub ID range for containers:/var/home/containers:/sbin/nologin
```

We are including sssd-2.6.2-3.el8.x86_64 as part of RHCOS.

But the RPM scriptlets are being run as part of the `rpm-ostree compose` process on the server side as part of build process, so that might be where the disconnect is.

We previously haven't encountered this issue with older versions of `sssd`.  We moved from RHEL 8.4 EUS to RHEL 8.6 Beta when this problem showed up.

Specifically, `sssd 2.5.2-2.el8_5.4 -> 2.6.2-3.el8`

Let me reach out to more folks in the CoreOS team about this.

RHCOS uses nss-altfiles to separate out system users into /usr/lib from local users in /etc. Looking at an RHCOS 8.6 pipeline build, we do have the sssd user and group:

[root@cosa-devsh ~]# grep sssd /usr/lib/passwd
sssd:x:995:993:User for sssd:/:/sbin/nologin
[root@cosa-devsh ~]# grep sssd /usr/lib/group
sssd:x:993:
[root@cosa-devsh ~]# getent passwd sssd
sssd:x:995:993:User for sssd:/:/sbin/nologin
[root@cosa-devsh ~]# getent group sssd
sssd:x:993:

Btw, take a note, that if the only SSSD domain backend running is 'implicit_files' (`sssd_be --domain implicit_files`) than an option could be to just disable SSSD by default (https://access.redhat.com/solutions/6815101), leaving the option to configure it explicitly if network identities are needed on the node.

(In reply to Jonathan Lebon from comment #9)
> RHCOS uses nss-altfiles to separate out system users into /usr/lib from
> local users in /etc. Looking at an RHCOS 8.6 pipeline build, we do have the
> sssd user and group:
> 
> [root@cosa-devsh ~]# grep sssd /usr/lib/passwd
> sssd:x:995:993:User for sssd:/:/sbin/nologin


Ah, this explains:

(In reply to Micah Abbott from comment #8)
> 
> We previously haven't encountered this issue with older versions of `sssd`. 
> We moved from RHEL 8.4 EUS to RHEL 8.6 Beta when this problem showed up.
> 
> Specifically, `sssd 2.5.2-2.el8_5.4 -> 2.6.2-3.el8`


The reason is https://github.com/SSSD/sssd/pull/5867  --  it was released upstream in sssd-2.6.2

AFAIU, upstream did not spot this one due to this change: https://fedoraproject.org/wiki/Changes/FlexibleLocalUserCache

Upstream PR: https://github.com/SSSD/sssd/pull/6108

Pushed PR: https://github.com/SSSD/sssd/pull/6108

* `master`
    * 3c6218aa91026e066e793ee26333ea64fd6bc50e - Revert "man: sssd.conf and sssd-ifp clarify user option"
    * 37f90057792a0b4543f34684ed9a240fe8e869c1 - Revert "usertools: force local user for sssd process user"

Maybe it is missing Verified:Tested flag. 
@shridhar, could you help to do some testing based on the new build? And I can help to test from my side.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7739