Bug 1398696

Summary: system-upgrade reboot loop when selinux is enforcing
Product: [Fedora] Fedora Reporter: Sam Varshavchik <mrsam>
Component: dnf-plugin-system-upgradeAssignee: Will Woods <wwoods>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 24CC: wwoods, yves.lecuyer.linfedora, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-13 00:31:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Sam Varshavchik 2016-11-25 15:48:37 UTC
Description of problem:

'dnf system-upgrade reboot' gets stuck in an infinite reboot loop when selinux is enforcing.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

On Fedora 24:

dnf system-upgrade download --releasever 25
dnf system-upgrade.reboot

Actual results:

The upgrade never starts. Infinite reboot loop.

Expected results:

Upgrade to Fedora 25

Additional info:

After booting to an emergency shell, examination of journalctl -b -1 uncovered the following relevant bits:

Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: Starting Bluetooth service...
Nov 25 09:51:55 thinkpenguin.email-scan.com bluetoothd[803]: Bluetooth daemon 5.43
Nov 25 09:51:55 thinkpenguin.email-scan.com audit[1]: AVC avc:  denied  { open } for  pid=1 comm="systemd" path="/var/lib/dnf/system-upgrade/.dnf-system-upgrade" dev="dm-1" ino=1181602 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=0
Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: dnf-system-upgrade.service: Failed to load environment files: Permission denied
Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: dnf-system-upgrade.service: Failed to run 'start' task: Permission denied
Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: Failed to start System Upgrade.
Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: dnf-system-upgrade.service: Unit entered failed state.
Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: dnf-system-upgrade.service: Failed with result 'resources'.
Nov 25 09:51:55 thinkpenguin.email-scan.com systemd[1]: Rebooting as result of failure.

This triggered a reboot, which simply failed the same way, etc...

Setting

SELINUX=permissive

in /etc/selinux/config, and rebooting, allowed the upgrade to 25 to succeed.

Comment 1 Sam Varshavchik 2016-11-25 18:02:59 UTC
After additional investigation:

# ls -alZd /var/lib/dnf/system-upgrade
drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0 233472 Nov 25      ↩
10:31 /var/lib/dnf/system-upgrade

After rmdir-ing and mkdir-ing the now-empty directory, its context is now:

drwxr-xr-x. 6 root root unconfined_u:object_r:rpm_var_lib_t:s0 4096 Nov 25 10:47 /var/lib/dnf

which seems to be what it should be. It appears that the original selinux context was wrong. It's unclear why the context was wrong, but 'system-upgrade reboot' didn't complain, and just forged ahead.

I'm thinking that 'system-upgrade reboot' should sanity-check the selinux context of all files, or at least the selinux context of the directory, before pulling the trigger.

Comment 2 Zbigniew Jędrzejewski-Szmek 2016-12-13 00:20:43 UTC
*** Bug 1403601 has been marked as a duplicate of this bug. ***

Comment 3 Zbigniew Jędrzejewski-Szmek 2016-12-13 00:26:26 UTC
> I'm thinking that 'system-upgrade reboot' should sanity-check the selinux context of all files, or at least the selinux context of the directory, before pulling the trigger.

Nah, system-upgrade is rather simple. SELinux is just too complicated, I don't think we can check if things will work without running the reboot.

In https://github.com/systemd/systemd/commit/953bf4604f I added an additional unit which should guard against reboot loops. So in principle you should just get a double reboot and boot back into the old system, where you can look at logs and maybe adjust selinux permissions or whatever and restart the upgrade.

Comment 4 Zbigniew Jędrzejewski-Szmek 2016-12-13 00:31:51 UTC

*** This bug has been marked as a duplicate of bug 1349721 ***