Bug 2215335

The ssh logins a failing to look up authorized keys due to SELinux. Whether this is causing login to fail altogether I can't say. 

type=AVC msg=audit(1685994998.957:300): avc:  denied  { read } for  pid=1848 comm="sshd" name="authorized_keys" dev="dm-1" ino=8388739 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0

There are quite a few other SELinux issues reported, though probably unrelated to ipa-client/SSSD.

The SSSD log is reporting issues with DNS:
(0x0040): [RID#1] SRV query failed [11]: Could not contact DNS servers

along with a few other failed DNS queries for the IPA server.

Then it fails to connect to LDAP and the domain is marked offline.

Those are the things that jump out to me.

authorized_keys file is under user and admin control. If SELinux labels are wrong, this is a configuration issue. None of IPA configuration code touches authorized_keys file, so this problem is not really IPA related.

I'd suggest to close this bug as NOTABUG. If after fixing wrong labels things still don't work, please start by opening a bugzilla with more specific details against SELinux first.

A few things of note. 
While I agree that authorized_keys having an improper selinux context could cause remote logins to fail, i would not expect them to fail in this way.

Selinux preventing the read of my authorized_keys would result in a permission denied, and then ssh would fall back to another mechanism.  If no other mechanism is enabled, it would simply reject my remote login.  

In this case, attempting a remote login triggers a  resource consumption loop, where you'll see memory get chunked away until the system is left at 100% memory consumption.  This does not feel like intended behavior. 

Also, my ssh keys are managed by IDM, not files on my filesystem.

Also, this exact same behavior is witnessed when logging in directly, with a username and password, at the vm's console, where no ssh keys are involved.

type=AVC msg=audit(1685991756.260:54): avc:  denied  { read } for  pid=1150 comm="sshd" name="authorized_keys" dev="dm-1" ino=8388737 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0

This AVC is from "sosreport before IMA was enabled" so it should not be related to IMA and it it looks like the system was already mis-labeled (in sense of SELinux) after "1. start with an idm joined rhel 9.2 system".

"unlabeled_t" usually means that the file was created on a system with disabled SELinux.

Are you able to reproduce this on a fresh system?
Are you able to reproduce it after you do the following steps before or after "1. start with an idm joined rhel 9.2 system"?

1. switch system to SELinux to permissive - set SELINUX=permissive in /etc/selinux/config
2. autorelabel the system - fixfiles -F onboot; reboot
3. switch the system back to enforcing - SELINUX=enforcing in /etc/config/selinux
4. reboot

(In reply to Petr Lautrbach from comment #5)
> type=AVC msg=audit(1685991756.260:54): avc:  denied  { read } for  pid=1150
> comm="sshd" name="authorized_keys" dev="dm-1" ino=8388737
> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0
> 
> This AVC is from "sosreport before IMA was enabled" so it should not be
> related to IMA and it it looks like the system was already mis-labeled (in
> sense of SELinux) after "1. start with an idm joined rhel 9.2 system".
> 
> "unlabeled_t" usually means that the file was created on a system with
> disabled SELinux.
> 
> Are you able to reproduce this on a fresh system?
> Are you able to reproduce it after you do the following steps before or
> after "1. start with an idm joined rhel 9.2 system"?
> 
> 1. switch system to SELinux to permissive - set SELINUX=permissive in
> /etc/selinux/config
> 2. autorelabel the system - fixfiles -F onboot; reboot
> 3. switch the system back to enforcing - SELINUX=enforcing in
> /etc/config/selinux
> 4. reboot

I can test. 

I will add that these were freshly built RHEL systems, in my lab.  They had no history, selinux has been enabled and enforcing since install.  If the filesystem is mislabeled, its either our install that did it, or the idm client install.

The documentation for IMA enablement explicitly says:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/enhancing-security-with-the-kernel-integrity-subsystem_managing-monitoring-and-updating-the-kernel#enabling-integrity-measurement-architecture-and-extended-verification-module_enhancing-security-with-the-kernel-integrity-subsystem (step 12)
---------
Warning

Enabling IMA and EVM without relabeling the system might make the majority of the files on the system inaccessible. 
---------

Did you relabel the system before continuing? 

I think at this point we should wait for your answers. 

If IMA/EVM was enabled after IPA enrollment, IPA is not involved in the issues you are experiencing unless IPA installer triggers creation of files that have no labels in the resulting SELinux policy.

So I'll put NEEDINFO on the reporter due to this and comment 6.

One point of observation.  During the process of enabling IMA, this failure occurred at step 2.  The notice about relabeling isn't until after step 12.  I never got that far.  But, aside from that. I just tested out relabeling on my att1 system that I used to collect the earlier sosreports. I will attach more sosreports after this description. 

I booted att1 by removing the "ima_policy=appraise_tcb ima_appraise=fix evm=fix" kernel args at boot time by modifying the grub boot entry
After booting, I did the following:
Logged in using my idm credentials. 
sudo touch ./autorelabel
reboot

While booting, I again removed the "ima_policy=appraise_tcb ima_appraise=fix evm=fix" at boot time by modifying the grub boot entry (as I expect the system will still not boot properly if I do not)
I let the system boot, it performed its relabel, and rebooted on its own at the end. 

I again removed "ima_policy=appraise_tcb ima_appraise=fix evm=fix" during the boot process so I was sure I could login and create an sosreport before testing but after performing the relabel.  That sosreport will be attached as:

sosreport-att1-bz2215335-2023-08-14-rurvweh_after-relabel-before-ima.tar.xz

After collecting the sosreport, I rebooted, and let the system boot with the "ima_policy=appraise_tcb ima_appraise=fix evm=fix" in place. 
after the system completed its boot, I attempted to login via the console (as I was at the console anyway) and encountered the exact same memory exhaustion issue that I had before attempting the relabel. 

I then forced a reboot, as the system was unresponsive, again removed "ima_policy=appraise_tcb ima_appraise=fix evm=fix" at boot time by modifying the grub boot entry, and collected a second sosreport after the system booted. That sosreport will be attached as

sosreport-att1-bz2215335-2023-08-14-ryxiqyy_after-relabel-after-ima-failure.tar.xz



I have not yet performed the actions suggested in comment #5, so I will keep the needinfo flag until I have also tried that.

ok, I have completed the request in commend #5. Here is my log:

---
Made a new vm, we'll call him att4
These libvirt systems are bios by default, so after the initial build I virsh edit'ed it to uefi (with secure boot disabled as the ima enable instructions say to do). i mention this only in case it may matter somehow.

My vm's come from a base image, that comes from the insights image builder.  I built a brand new image for this test (Following my usual standards for disk layout, and some packages I like to have installed), in case there maybe was some imagebuilder bug contributing to the issue.

NOTE: Unless otherwise noted, I am attempting my "test" to see if I have the issue by logging into the vm console directly (not over ssh) with my idm user.  That user's name is 'gangrif'



joined new system to idm domain using the ansible-freeipa module. (this is how I do all of my lab systems that will be idm clients)

Installed all updates (nothing to install)

performed the pre-checks in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/enhancing-security-with-the-kernel-integrity-subsystem_managing-monitoring-and-updating-the-kernel#enabling-integrity-measurement-architecture-and-extended-verification-module_enhancing-security-with-the-kernel-integrity-subsystem

[root@att4 ~]# mount | grep securityfs
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
[root@att4 ~]# dmesg | grep -i -e EVM -e IMA
... snip ...
[    2.965538] systemd[1]: systemd 252-14.el9_2.3 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    3.690851] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[    7.210946] systemd[1]: systemd 252-14.el9_2.3 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)


Performed step 1 of the ima setup procedure:
grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="ima_policy=appraise_tcb ima_appraise=fix evm=fix"

Then rebooted as Step 2 instructs.


Experienced the same memory exhaustion issue.

Rebooted without ima (by removing the kernel command line args in grub at boot time)
set selinux=permissive (it was set to enforcing, as expected) in /etc/selinux/config

rebooted, with ima enabled, to see if simply leaving selinux permissive would change anything. I experienced the same memory exhaustion issue

Rebooted again, turning off ima at boot time in grub.
logged in, double-checked with getenforce, selinux is in fact permissive.

[gangrif@att4 ~]$ getenforce
Permissive

ran: fixfiles -F onboot, rebooted

It was not clear whether I should have the ima kernel args in place after reboot, so I assumed that we'd like the system to be in a known good state when doing so, so I rebooted, and again disabled ima during boot by removing the kernel args that enable it in grub. Let the sytem do its relabel. The system auto-reboots after the relable, I let it fully boot after that WITHOUT ima enabled.

After the system booted, I set selinux back to enforcing, and rebooted (this time leaving the ima kernel args in place).

after the reboot, i again logged in via the console with my idm user.

The problem persists.  cannot login with idm credentials, system goes into a memory consumption loop eventually consuming all free memory, and 1 entire cpu. This is according to the memory and cpu usage in the rhel web console's virtual machine manager, i have no way to see the actual usage within the vm, as it is un-usable

Rebooted, removing ima at boot time, and collected sosreport-att4-bz2215335-2023-08-14-ixzvvcl.tar.xz (which I will attach shortly)