Bug 1349721 - Upgrade from Fedora 24 to Fedora Rawhide with dnf-plugin-system-upgrade fails, system stuck in boot loop until booted with enforcing=0
Summary: Upgrade from Fedora 24 to Fedora Rawhide with dnf-plugin-system-upgrade fails...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy
Version: 25
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: ---
Assignee: Lukas Vrabec
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1398696 (view as bug list)
Depends On:
Blocks: F25BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2016-06-24 05:33 UTC by Adam Williamson
Modified: 2016-12-13 00:31 UTC (History)
10 users (show)

Fixed In Version: selinux-policy-3.13.1-208.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-17 03:04:37 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2016-06-24 05:33:00 UTC
To reproduce, just try to upgrade from Fedora 24 to current Rawhide with dnf-plugin-system-upgrade. When you run `dnf system-upgrade reboot`, the system will get stuck in an eternal boot loop - it boots, fails to reach the upgrade target, reboots, tries again, ad infinitum. You can break out by booting with `enforcing=0`, which allows the system to reach the upgrade target and successfully upgrade.

Strangely, this doesn't seem to affect upgrades from F23 to Rawhide, those work fine.

Proposing as a Beta blocker: "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

I saw this when upgrading my desktop to Rawhide, openQA is running into it - see https://openqa.fedoraproject.org/tests/23677 , for e.g. - and nivag reported the same thing today in IRC.

Comment 1 Adam Williamson 2016-06-24 19:32:49 UTC
So a bit more detail here...to debug this, you can copy /usr/lib/systemd/system/dnf-system-upgrade.service to /etc/systemd/system and drop the FailureAction=reboot line from the copy; this will prevent the system going into a boot loop. You can also do `systemctl enable debug-shell.service` , which gets you a shell on tty9.

Armed with that, and systemd.log_level=debug, I can see this:

dnf-system-upgrade.service: ConditionPathExists=/system-update/.dnf-system-upgrade succeeded.
dnf-system-upgrade.service: Failed to load environment files: Permission denied
dnf-system-upgrade.service: Failed to run 'start' task: Permission denied
dnf-system-upgrade.service: Changed dead -> failed
dnf-system-upgrade.service: Job dnf-system-upgrade.service/start finished, result=failed

I can't find any trace of an AVC, but it's certainly an SELinux issue, since it works fine with enforcing=0.

/system-updates/.dnf-system-upgrade itself seems to be the 'EnvironmentFile' that systemd wants to open - it's listed as EnvironmentFile in the unit - and its permissions are:

-rw-r--r--. 1 root.root unconfined_u:object_r:rpm_var_lib_t:s0

It would be systemd itself which would try to read the file, I believe, and which is presumably being denied.

Comment 2 Adam Williamson 2016-06-24 20:39:40 UTC
I checked the F23 case; /system-update/.dnf-system-upgrade has the exact same permissions and label there, so I guess the difference is that in F23 systemd is permitted to read that file but in F24 it is not...

Comment 3 Zbigniew Jędrzejewski-Szmek 2016-07-19 20:40:55 UTC
I think that the solution is the same as in the (unfinished) .autorelabel discussion: we should boot selinux in permissive mode for the upgrade.

Comment 4 Petr Schindler 2016-07-21 09:39:14 UTC
Discussed at 2016-07-20 blocker review meeting: [1]. 

This bug was accepted as Beta blocker: Failing upgrade violates the beta criterion: "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-07-20/f25-blocker-review.2016-07-20-16.00.html

Comment 5 Petr Lautrbach 2016-07-21 14:35:36 UTC
The reason why it fails is that systemd running as init_t tries to read EnvironmentFile - /system-update/.dnf-system-upgrade which is labeled with rpm_var_lib_t type and it's not allowed in the policy:


# sesearch -A -s init_t -t rpm_var_lib_t 
Found 2 semantic av rules:
   allow init_t file_type : dir { getattr search open } ; 
   allow init_t file_type : filesystem { unmount getattr } ; 


/system-update/ is symlink to /var/lib/dnf/systemd-upgrade which according to matchpathcon has a correct label:

# matchpathcon /var/lib/dnf/systemd-upgrade/.dnf-system-upgrade
/var/lib/dnf/systemd-upgrade/.dnf-system-upgrade	system_u:object_r:rpm_var_lib_t:s0

So the fix could be to add a policy which allows init_t read rpm_var_lib_t.

You can try the following example local policy module:

# cat > localdnfupgradefix.cil <<EOF
(allow init_t rpm_var_lib_t (file (getattr open read)))
EOF
# semodule -i localdnfupgradefix.cil
# dnf system-upgrade reboot

Comment 6 Lukas Vrabec 2016-07-21 14:53:19 UTC
It make sense. init_t domain was unconfined domain till F23. You can output from F23:
$ sesearch -A -s init_t -t rpm_var_lib_t -c file -p read 
Found 1 semantic av rules:
   allow files_unconfined_type file_type : file { ioctl read write create getattr setattr lock relabelfrom relabelto append unlink link rename execute swapon quotaon mounton execute_no_trans open audit_access } ;


This is reason why it was allowed in >F23. 

From F24, init_t is not unconfined_domain and there is no allow rule for this:

# sesearch -A -s init_t -t rpm_var_lib_t -c file -p read
#

I believe we should add this rule to make it working. 

Petr, 
Thanks for testing.

Comment 7 Jan Kurik 2016-07-26 04:37:27 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle.
Changing version to '25'.

Comment 8 Fedora Update System 2016-08-12 15:57:50 UTC
selinux-policy-3.13.1-208.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-662487f8f1

Comment 9 Fedora Update System 2016-08-17 03:02:58 UTC
selinux-policy-3.13.1-208.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Lou Hafer 2016-11-24 16:31:57 UTC
It's still possible to find oneself stuck in this loop using the --datadir option offered by dnf-plugin-system-upgrade. Just upgraded from Fed24 to Fed25 and relocated datadir under /home because it had lots of space. Result was SEL type of home_root_t and failure as described above. The description above is very clear and I was able to use a Fed24 Live CD to boot the system, mount the relevant logical volume, and use chcon to change the SEL type to rpm_var_lib_t. The system upgrade completed with no further issues.

This is a bit of a trap --- there's no hint that using --datadir will get you this sort of grief. From a user perspective the ideal fix would be for dnf-plugin-system-upgrade to force the SEL context it requires, but this might not be ideal from a security standpoint. Next best would be a warning, something like `You've moved the data directory; ensure that the SEL context is rpm_var_lib_t before proceeding with reboot.'

Comment 11 Zbigniew Jędrzejewski-Szmek 2016-12-13 00:31:51 UTC
*** Bug 1398696 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.