Bug 2069102

Summary: failed boot with selinux set to enforcing
Product: [Fedora] Fedora Reporter: Simon Putt <lemonzest>
Component: selinux-policyAssignee: Zdenek Pytela <zpytela>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: medium    
Version: 36CC: dominik, dwalsh, grepl.miroslav, johannes.kalliauer, lvrabec, mmalik, omosnace, otto.liljalaakso, pkoncity, vmojzis, zpytela
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
If this bug requires documentation, please select an appropriate Doc Type value.
Last Closed: 2022-07-08 15:05:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journal log of failed boot
none
failed logins
none
audit log
none
result of the restorecon command
none
some failures when reinstalling selinux packages
none
journalctl -xb --no-pager|grep -i denied none

Description Simon Putt 2022-03-28 09:43:31 UTC
Created attachment 1868679 [details]
journal log of failed boot

Created attachment 1868679 [details]
journal log of failed boot

Description of problem:

System will not boot with selinux set to enforcing

Version-Release number of selected component (if applicable):

selinux-policy-36.5-1.fc36.src.rpm (but also unknown, I'm making a guess here)

How reproducible:

always, when I set /etc/sysconfig/selinux to enforcing it results in no boot

Steps to Reproduce:
1. upgrade to latest fedora 36 beta
2. reboot
3.

Actual results:

no boot

Expected results:

system to boot

Additional info:

Comment 1 Zdenek Pytela 2022-03-28 15:21:56 UTC
Thanks for the report, I can't find any data which would help with troubleshooting though, like AVC denials. If auditd is running, can you attach audit log or output of

  # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today
and
  # rpm -qa "selinux-policy*"

There are some failing services, including selinux-autorelabel-mark.service, but I can't assess if your system needs relabeling.

Comment 2 Simon Putt 2022-03-29 12:57:22 UTC
Created attachment 1868968 [details]
failed logins

Comment 3 Simon Putt 2022-03-29 12:59:29 UTC
Created attachment 1868969 [details]
audit log

Comment 4 Simon Putt 2022-03-29 13:00:13 UTC
selinux-policy-36.5-1.fc36.noarch
selinux-policy-targeted-36.5-1.fc36.noarch
selinux-policy-sandbox-36.5-1.fc36.noarch

Comment 5 Zdenek Pytela 2022-03-29 16:44:14 UTC
These audit records indicate mislabeled system:

Mar 29 13:32:35 electronicbox audit[2559]: AVC avc:  denied  { transition } for  pid=2559 comm="gdm-session-wor" path="/etc/gdm/PreSession/Default" dev="nvme0n1p3" ino=9569691 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0
Mar 29 13:32:35 electronicbox gdm-password][2530]: Gdm: Unable to run script: Failed to execute child process “/etc/gdm/PreSession/Default” (Permission denied)

Mar 29 13:32:34 electronicbox audit[2541]: AVC avc:  denied  { transition } for  pid=2541 comm="(systemd)" path="/usr/lib/systemd/systemd" dev="nvme0n1p3" ino=12211671 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0

What is the SELinux type of systemd?

  # ls -lZ /usr/lib/systemd/systemd
-rwxr-xr-x. 1 root root system_u:object_r:init_exec_t:s0 1601288 25. led 13.31 /usr/lib/systemd/systemd

What is the output of, e. g.

  # restorecon -Rvn /usr/lib

With the -n switch no change applies.

Anyway I'd suggest to relabel all files on filesystems which can take a long time though, depending on the files number:

  # fixfiles -F onboot
  # reboot

Comment 6 Simon Putt 2022-03-30 02:41:29 UTC
-rwxr-xr-x. 1 root root system_u:object_r:lib_t:s0 1983568 Mar 18 13:01 /usr/lib/systemd/systemd

Odd tho, first thing I did after upgrading WAS to do a relable, suppose another won't hurt, will report back.

Comment 7 Simon Putt 2022-03-30 03:07:21 UTC
Done 2 replables and its not helping it login, and when I re enable permissive and logged in and  did the "restorecon -Rvn /usr/lib" line it's still the same, says some things would be relabled but after the relabel they are still not labeled correctly

Comment 8 Simon Putt 2022-03-30 03:09:17 UTC
Created attachment 1869230 [details]
result of the restorecon command

Comment 9 Simon Putt 2022-03-30 06:38:34 UTC
Created attachment 1869242 [details]
some failures when reinstalling selinux packages

Comment 10 Zdenek Pytela 2022-03-30 07:04:27 UTC
> Would relabel /usr/lib/libmpeg2.so.0.1.0 from system_u:object_r:lib_t:s0 to system_u:object_r:textrel_shlib_t:s0

With the -n switch no change applies, run restorecon -Rv or fixfiles.

Comment 11 Simon Putt 2022-03-30 07:10:47 UTC
(In reply to Zdenek Pytela from comment #10)
> > Would relabel /usr/lib/libmpeg2.so.0.1.0 from system_u:object_r:lib_t:s0 to system_u:object_r:textrel_shlib_t:s0
> 
> With the -n switch no change applies, run restorecon -Rv or fixfiles.

Should I run restorecon on the whole filesystem or just /usr/lib as I already done the later and it threw up more errors in the boot than before and results in no login, I have also ran fixfiles -F onboot twice now and same results, should I run sudo restorecon -Rv / next?

Comment 12 Zdenek Pytela 2022-03-30 08:07:06 UTC
(In reply to Simon Putt from comment #11)
> (In reply to Zdenek Pytela from comment #10)
> > > Would relabel /usr/lib/libmpeg2.so.0.1.0 from system_u:object_r:lib_t:s0 to system_u:object_r:textrel_shlib_t:s0
> > 
> > With the -n switch no change applies, run restorecon -Rv or fixfiles.
> 
> Should I run restorecon on the whole filesystem or just /usr/lib as I
> already done the later and it threw up more errors in the boot than before
> and results in no login, I have also ran fixfiles -F onboot twice now and
> same results, should I run sudo restorecon -Rv / next?

I am puzzled now, fixfiles should be sufficient to run once. If the system is seriously mislabeled, you may need to boot in SELinux permissive mode which changes in /etc/selinux/config

Then

  # fixfiles -F onboot
  # reboot

should relabel and no other action should be needed other than changing SELinux to enforcing.

Comment 13 Otto Liljalaakso 2022-03-30 08:22:45 UTC
I suffer from similar problems after F36 update,
not possible to login while SELinux is enforcing,
caused by widespread SELinux mislabelling.

For me, 'fixfiles -F onboot && reboot' did not fix the problem,
but 'restorecon -Rv / && reboot' did.
SELinux was permissive during both of those.

Is that a bug in the fixfiles utility then?
Where should it be reported?

Comment 14 Simon Putt 2022-03-30 08:29:20 UTC
In the past I've done exactly that! and in worked but on this occasion it does not, dunno why, however I did "sudo restorecon -Rv /*" and it relabeled TONS of stuff allowing me to login after boot with enforcing enabled, however tho some container runtimes did not relabel and I have this in my dmes log

[  249.069588] SELinux:  Context unconfined_u:object_r:vnc_home_t:s0 is not valid (left unmapped).
[  353.016694] SELinux:  Context system_u:object_r:conmon_exec_t:s0 is not valid (left unmapped).
[  353.019977] SELinux:  Context system_u:object_r:swtpm_exec_t:s0 is not valid (left unmapped).
[  353.021751] SELinux:  Context system_u:object_r:osbuild_exec_t:s0 is not valid (left unmapped).
[  414.130336] SELinux:  Context system_u:object_r:flatpak_helper_exec_t:s0 is not valid (left unmapped).
[  414.131554] SELinux:  Context system_u:object_r:vnc_session_exec_t:s0 is not valid (left unmapped).
[  422.445613] SELinux:  Context system_u:object_r:container_log_t:s0 is not valid (left unmapped).

the container runtimes affected are, lxc, lxd and /dev/tpm stuff.

[FAILED] Failed to start lxd-containers.service - LXD - container startup/shutdown.
See 'systemctl status lxd-containers.service' for details.
[FAILED] Failed to start tpm2-abrmd.service - TPM2 Access Broker and Resource Management Daemon.
See 'systemctl status tpm2-abrmd.service' for details.
[FAILED] Failed to start lxc-net.service - LXC network bridge setup.
See 'systemctl status lxc-net.service' for details.
[FAILED] Failed to start lxc.service - LXC Container Initialization and Autoboot Code.
See 'systemctl status lxc.service' for details.

Comment 15 Simon Putt 2022-03-30 08:31:52 UTC
Created attachment 1869283 [details]
journalctl -xb --no-pager|grep -i denied

Comment 16 Otto Liljalaakso 2022-03-30 22:31:12 UTC
I did some more research on my system with this problem.
I seems that 'fixfiles -F onboot && reboot' actually breaks logging in with selinux enabled.
I can reproduce this reliably on my system:

1. Start with a system where login works, selinux is enforcing, but suffering from other selinux related problems.
2. Try to fix the problems with 'fixfiles -F onboot && reboot'.
3. Observe the following suspicious output during relabel:

    selinux-autorelabel[1212]: Failed to resolve allow statement at /var/lib/selinux/targeted/tmp/modules/200/container/cil:1263
    selinux-autorelabel[1212]: Failed to resolve AST
    selinux-autorelabel[1212]: genhomedircon:  Failed!
    selinux-autorelabel[1215]: /sbin/setfiles: /etc/selinux/targeted/contexts/files/file_contexts.bin: context system_u:object_r:container_runtime_exec_t:s0 is invalid
    selinux-autorelabel[1215]: /sbin/setfiles: /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin: context unconfined_u:object_r:snappy_home_t:s0 is invalid
    selinux-autorelabel[1215]: /sbin/setfiles: conflicting specifications for /usr/bin/qmake-qt5 and /usr/lib64/qt5/bin/qmake, using system_u:object_r:lib_t:s0.
    (many more "conflicting specifications")

4. Try to login when the login screen opens, observe that login attempts throw you back to login screen
5. Via a rescue usb stick, switch selinux to permissive
6. Reboot, observe that login works
7. Do 'restorecon -vR /', observe very large amount of relabelling.
   The following is just an example, it is not just /var/lib and not just mock,
   but the common theme seems to bethat the third part of the label changes from generic to specific
   (var_t to mock_var_lib_t in this example, the other lines look similar):

    Relabeled /var/lib/mock/fedora-rawhide-i686-bootstrap/root/usr/share/locale/ia from system_u:object_r:var_t:s0 to system_u:object_r:mock_var_lib_t:s0

8. Set selinux to enforcing, reboot
9. Observe that login works, everything seems to be back in step 1

Comment 17 Simon Putt 2022-03-30 22:34:35 UTC
(In reply to Otto Urpelainen from comment #16)
> I did some more research on my system with this problem.
> I seems that 'fixfiles -F onboot && reboot' actually breaks logging in with
> selinux enabled.
> I can reproduce this reliably on my system:
> 
> 1. Start with a system where login works, selinux is enforcing, but
> suffering from other selinux related problems.
> 2. Try to fix the problems with 'fixfiles -F onboot && reboot'.
> 3. Observe the following suspicious output during relabel:
> 
>     selinux-autorelabel[1212]: Failed to resolve allow statement at
> /var/lib/selinux/targeted/tmp/modules/200/container/cil:1263
>     selinux-autorelabel[1212]: Failed to resolve AST
>     selinux-autorelabel[1212]: genhomedircon:  Failed!
>     selinux-autorelabel[1215]: /sbin/setfiles:
> /etc/selinux/targeted/contexts/files/file_contexts.bin: context
> system_u:object_r:container_runtime_exec_t:s0 is invalid
>     selinux-autorelabel[1215]: /sbin/setfiles:
> /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin: context
> unconfined_u:object_r:snappy_home_t:s0 is invalid
>     selinux-autorelabel[1215]: /sbin/setfiles: conflicting specifications
> for /usr/bin/qmake-qt5 and /usr/lib64/qt5/bin/qmake, using
> system_u:object_r:lib_t:s0.
>     (many more "conflicting specifications")
> 
> 4. Try to login when the login screen opens, observe that login attempts
> throw you back to login screen
> 5. Via a rescue usb stick, switch selinux to permissive
> 6. Reboot, observe that login works
> 7. Do 'restorecon -vR /', observe very large amount of relabelling.
>    The following is just an example, it is not just /var/lib and not just
> mock,
>    but the common theme seems to bethat the third part of the label changes
> from generic to specific
>    (var_t to mock_var_lib_t in this example, the other lines look similar):
> 
>     Relabeled
> /var/lib/mock/fedora-rawhide-i686-bootstrap/root/usr/share/locale/ia from
> system_u:object_r:var_t:s0 to system_u:object_r:mock_var_lib_t:s0
> 
> 8. Set selinux to enforcing, reboot
> 9. Observe that login works, everything seems to be back in step 1

This is exactly what I observed also

Comment 18 Dominik 'Rathann' Mierzejewski 2022-03-31 20:28:17 UTC
I had exactly the same issue and despite removing both flatpak-selinux and containers-selinux I still had the corresponding selinux modules:
# semodule -lfull |grep container
200 container            pp          
# semodule -lfull |grep flatpak
200 flatpak              pp          
# semodule -X 200 -r flatpak
libsemanage.semanage_direct_remove_key: Removing last flatpak module (no other flatpak module exists at another priority).
Failed to resolve allow statement at /var/lib/selinux/targeted/tmp/modules/200/container/cil:1263
Failed to resolve AST
semodule:  Failed!
# semodule -X 200 -r container
libsemanage.semanage_direct_remove_key: Removing last container module (no other container module exists at another priority).
Failed to resolve allow statement at /var/lib/selinux/targeted/tmp/modules/200/flatpak/cil:122
Failed to resolve AST
semodule:  Failed!

On a hunch, I tried removing them both at the same time:

# semodule -X 200 -r container -r flatpak
libsemanage.semanage_direct_remove_key: Removing last container module (no other container module exists at another priority).
libsemanage.semanage_direct_remove_key: Removing last flatpak module (no other flatpak module exists at another priority).
# semodule -lfull |grep -E 'flatpak|container'
#

It seems it worked. Hopefully the above helps.

Comment 19 Simon Putt 2022-04-02 07:28:30 UTC
The above did not work for me

Comment 20 Zdenek Pytela 2022-04-14 17:16:04 UTC
F35->F36 upgrades currently may fail when there are packages with custom SELinux modules installed, like container, flatpak, osbuild. We are working on resolving problems like this.

Problem with fixfiles not relabeling properly on boot seems to be a different one though.

Comment 21 Zdenek Pytela 2022-07-08 15:05:16 UTC
As there is not enough information to investigate on the issue for an extended period of time, I am going to close this bug due to inactivity. If you would like to pursue this matter, feel free to reopen this bug and attach the needed information.

Comment 22 Otto Liljalaakso 2022-07-11 08:48:03 UTC
I cannot reproduce the issue using my instructions in comment 14 anymore.
I am not sure if it is because my system does not have any selinux issues anymore,
or because 'fixfiles -F onboot' has changed in the meantime.
Be that as it may, from my point of view, this is issue is resolved.