Bumping priority, as PAM regressions that prevent logging in are quite some blocker. This doesn't happen in our upstream RHEL 8.3 images (pretty much bog standard virt-install from nightly), so I check how the OSCI gatingimage (no idea how it gets built) differs from our's. Both have pam-1.3.1-10.el8.x86_64 and systemd-pam-239-30.el8_2.x86_64 and no other packages matching "*pam*".

But the CI one uses authselect, while our's doesn't. On the broken image:

# cat /etc/authselect/authselect.conf 
sssd

And indeed when I run "authselect select minimal" the failures go away, and they come back after "authselect select sssd". The difference in system-auth:

--- /tmp/system-auth	2020-05-14 07:46:15.740078340 -0400
+++ /etc/pam.d/system-auth	2020-05-14 07:46:19.312122686 -0400
@@ -1,15 +1,24 @@
-# Generated by authselect on Thu May 14 07:46:07 2020
+# Generated by authselect on Thu May 14 07:46:19 2020
 # Do not modify this file manually.
 
 auth        required                                     pam_env.so
 auth        required                                     pam_faildelay.so delay=2000000
+auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
+auth        [default=1 ignore=ignore success=ok]         pam_localuser.so
 auth        sufficient                                   pam_unix.so nullok try_first_pass
+auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
+auth        sufficient                                   pam_sss.so forward_pass
 auth        required                                     pam_deny.so
 
 account     required                                     pam_unix.so
+account     sufficient                                   pam_localuser.so
+account     sufficient                                   pam_usertype.so issystem
+account     [default=bad success=ok user_unknown=ignore] pam_sss.so
+account     required                                     pam_permit.so
 
-password    requisite                                    pam_pwquality.so try_first_pass
+password    requisite                                    pam_pwquality.so try_first_pass local_users_only
 password    sufficient                                   pam_unix.so sha512 shadow nullok try_first_pass use_authtok
+password    sufficient                                   pam_sss.so use_authtok
 password    required                                     pam_deny.so
 
 session     optional                                     pam_keyinit.so revoke
@@ -17,3 +26,4 @@
 -session    optional                                     pam_systemd.so
 session     [success=1 default=ignore]                   pam_succeed_if.so service in crond quiet use_uid
 session     required                                     pam_unix.so
+session     optional                                     pam_sss.so

Of these, the culprit is pam_usertype -- if I comment out the three lines with it, the failure goes away. pam_usertype.so comes from the pam package itself, so the component is right.

I updated authselect this week to use pam_usertype so this is what's causing it. AFAIK pam_usertype.so was added in pam-1.3.1-9.el8 and authselect requires it. Can you check that the module is installed in /usr/lib64/security/pam_usertype.so?

Yes, it does exist, it just seems to misbehave:

/usr/lib64/security/pam_usertype.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=bbcaa4cf56438863069f5517ed2f271cf40af59f, stripped

Hmm. Same module works on Fedora for some time now. Diff between patch in Fedora and RHEL did not reveal anything that should cause it. I'll prep a machine and try to reproduce it.

Somehow I malformed the patch and I'm fixing it right now, I hope it doesn't take long to spread the new version of pam.

Verified the bug on PAM version: pam-1.3.1-11.el8.x86_64

Steps followed during verification:

1. Reproduce the issue with older pam version ie, pam-1.3.1-10.el8.x86_64

# dnf downgrade pam-1.3.1-10.el8.x86_64.rpm

# rpm -q pam
pam-1.3.1-10.el8.x86_64

2. Add a testuser to system.

$ id testuser
uid=1000(testuser) gid=1000(testuser) groups=1000(testuser)

3. Configure authselect with SSSD profile.

# authselect select --force sssd
# cat /etc/authselect/authselect.conf 
sssd

4. Run "su - testuser" and monitor /vag/log/secure file.

# tail -f /var/log/secure
.
.
Jul 20 03:28:52 martini_r83 su[2966]: PAM unable to resolve symbol: pam_sm_authenticate
Jul 20 03:28:52 martini_r83 su[2966]: PAM unable to resolve symbol: pam_sm_setcred
Jul 20 03:28:52 martini_r83 su[2966]: PAM unable to resolve symbol: pam_sm_acct_mgmt
Jul 20 03:28:52 martini_r83 su[2966]: pam_systemd(su-l:session): Cannot create session: Already running in a session or user slice
Jul 20 03:28:52 martini_r83 su[2966]: pam_unix(su-l:session): session opened for user testuser by root(uid=0)
Jul 20 03:29:12 martini_r83 su[2966]: pam_unix(su-l:session): session closed for user testuser
.
.

5. Observe the pam errors, these should not be preset.

6. Upgrade pam to latest version : pam-1.3.1-11.el8.x86_64

7. Repeat step(4).

Jul 20 03:31:33 martini_r83 su[3083]: pam_systemd(su-l:session): Cannot create session: Already running in a session or user slice
Jul 20 03:31:33 martini_r83 su[3083]: pam_unix(su-l:session): session opened for user testuser by root(uid=0)


8. With the latest build, the pam errors disappear.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pam bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4466