Bug 2304328

I bricked a few Silverblue/Kinoite systems while updating to the 20240810 rawhide OSTree image, which included a transition from ostree-boot to bootupctl (bootupd v0.2.20). I was able to recover the systems using a rescue image (copy EFI artifacts from a recent deployment to /boot/efi) but encountered a lot of issues along the way.

1. Secure boot/FDE fails during boot loader artifact updates. There doesn't appear to be a way to detect if a bootloader update like this one will boot correctly after the fact. In some cases I can boot using a backup LUKS key, but in this case I had to completely turn of secure boot in the BIOS.

2. The rescue image doesn't work. Is there a documented way to recover an OSTree system? Otherwise "recovery" in this case involved manually unlocking the root partition and digging around to find the correct deployment directory.

3. Updating boot artifacts via bootupd should be transactional. Once I got the systems to boot once with a bootupd-enabled image, running rpm-ostree a second time somehow made the EFI artifacts inconsistent; the system tried to boot the fedora EFI profile but failed and returned to the BIOS menu. Even better, bootupd should be parameterized to read artifacts from any deployment, not just the current one.

4. These systems were built starting with a Fedora 38 OSTree installer, which did not use a consistent partition naming scheme. From looking at the rust-bootupd sources I figured out that I could rename the ESP as "EFI-SYSTEM" and bootupctl would proceed further, but this is fragile. The partition search logic for bootupd should use the fstype UUID (c12a7328-f81f-11d2-ba4b-00a0c93ec93b) from e.g. 'parted print -j'

5. The SELinux policy for bootupd (bootupd.te in selinux-policy) is incomplete. The default label for /boot is boot_t and the default label for an ESP specified in /etc/fstab is dosfs_t, neither of which is readable by bootupd_t. If the ESP is pre-mounted to e.g. /boot/efi, bootupctl is able to find the partition but not read it. The bootupd_t type is not transitionable either, so I was not able to mount the ESP with '-o context='.

Aug 12 15:45:34 ce-link-94-c8-d5 systemd[1]: Started bootupd.service - /usr/bin/bootupctl -v2 status.
Aug 12 15:45:34 ce-link-94-c8-d5 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=bootupd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Aug 12 15:45:34 ce-link-94-c8-d5 audit[264814]: AVC avc:  denied  { getattr } for  pid=264814 comm="bootupctl" path="/boot/efi/EFI/BOOT/BOOTIA32.EFI" dev="nvme0n1p1" ino=171 scontext=system_u:system_r:bootupd_t:s0 tcontext=system_u:object_r:dosfs_t:s0 tclass=file permissive=0
Aug 12 15:45:34 ce-link-94-c8-d5 systemd[1]: bootupd.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 15:45:34 ce-link-94-c8-d5 systemd[1]: bootupd.service: Failed with result 'exit-code'.

Aug 12 16:12:15 ce-link-94-c8-d5 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=bootupd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Aug 12 16:12:15 ce-link-94-c8-d5 audit[265819]: AVC avc:  denied  { write } for  pid=265819 comm="bootupctl" path=2F626F6F742F233135202864656C6574656429 dev="nvme0n1p2" ino=15 scontext=system_u:system_r:bootupd_t:s0 tcontext=system_u:object_r:boot_t:s0 tclass=file permissive=0
Aug 12 16:12:15 ce-link-94-c8-d5 audit[265819]: AVC avc:  denied  { create } for  pid=265819 comm="bootupctl" name=".tmpQd6RdJ5a.tmp" scontext=system_u:system_r:bootupd_t:s0 tcontext=system_u:object_r:boot_t:s0 tclass=file permissive=0


6. SELinux policy gaps also prevented bootupctl from mounting the ESP dynamically. Presumably we need to borrow some of the logic from fstools.te to enable block device access.

Aug 12 16:43:30 ce-link-94-c8-d5 audit[266722]: AVC avc:  denied  { getattr } for  pid=266722 comm="bootupctl" path="/dev/nvme0n1p1" dev="devtmpfs" ino=445 scontext=system_u:system_r:bootupd_t:s0 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
Aug 12 16:43:30 ce-link-94-c8-d5 systemd[1]: bootupd.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 16:43:30 ce-link-94-c8-d5 systemd[1]: bootupd.service: Failed with result 'exit-code'.

7. The systemd-run logic in bootupctl is fragile and does not handle tool failures well. When bootupctl fails (like, because of SELinux issues) the failure is something cryptic like

Failed to start transient service unit: Unit bootupd.service failed to load properly, please adjust/correct and reload service manager: Device or resource busy

and if the transient unit does fail, it leaves a unit file in /run/systemd/transient preventing the tool from starting again.

I am concerned that we don't have a smooth upgrade path from F40 to F41 without fixes for some of these issues.



Reproducible: Always

Steps to Reproduce:
1. boot an OSTree system using bootupd
2. rm /boot/bootupd-state.json
3. setenforce 1
4. bootupctl update
Actual Results:  
bootupctl fails to update boot artifacts

Expected Results:  
bootupctl should either update the boot artifacts, or roll them back and give a useful error message

If the system is bricked, it can be recovered with some difficulty from a rescue image by overwriting /boot/efi/EFI/Boot and /boot/efi/EFI/fedora from the most recent deployment.

Otherwise, bootupd appears to run smoothly if selinux is disabled.

setenforce 0
bootupctl update