Bug 2247635

Hans coincidentally just hit the issue himself and says the problem is caused by boot_indeterminate not being set in grubenv during the offline update. He will investigate.

The issue seems to be that grub2-tools no longer ships the /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service symlink:

Working F38 system:

[hans@shalem ~]$ rpm -q grub2-tools
grub2-tools-2.06-95.fc38.x86_64
[hans@shalem ~]$ rpm -ql grub2-tools | grep grub-boot-indeterminate.service
/usr/lib/systemd/system/grub-boot-indeterminate.service
/usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service

Broken F39 system:

[root@fedora ~]# rpm -q grub2-tools
grub2-tools-2.06-104.fc39.x86_64
[root@fedora ~]# rpm -ql grub2-tools | grep grub-boot-indeterminate.service
/usr/lib/systemd/system/grub-boot-indeterminate.service

Note after installing recent updates on F38 it breaks there too:

[hans@shalem ipu6-drivers]$ rpm -q grub2-tools
grub2-tools-2.06-102.fc38.x86_64
[hans@shalem ipu6-drivers]$ rpm -ql grub2-tools | grep grub-boot-indeterminate.service
/usr/lib/systemd/system/grub-boot-indeterminate.service

Workaround (untested):

sudo ln -s ../grub-boot-indeterminate.service /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service

Fixing this should be a matter of just re-adding the accidentally dropped symlink to grub2-tools.

Asking the boot menu to be shown by e.g. keeping alt pressed when hitting the "restart" button on the restart? cancel/restart dialog in GNOME also no longer works.

The root cause is the same, in this case it is the:

/usr/lib/systemd/system/reboot.target.wants/grub2-systemd-integration.service

symlink which is missing in later grub builds.

Workaround:

sudo ln -s ../grub2-systemd-integration.service /usr/lib/systemd/system/reboot.target.wants/grub2-systemd-integration.service

And I just noticed a third issue due to another missing .wants symlink, sessions which last longer then 2 minutes used to count as boot-success, but they no longer do since the 
grub-boot-success.timer is no longer enabled by default.

Workaround:

sudo mkdir /usr/lib/systemd/user/timers.target.wants
sudo ln -s ../grub-boot-success.timer /usr/lib/systemd/user/timers.target.wants/grub-boot-success.timer

Looking at grub's package history this seems to be caused by the fixes for bug 2230575, which removed all 3 .wants symlinks. This is wrong for a variety of reasons:

1. For the /usr/lib/systemd/system/reboot.target.wants/grub2-systemd-integration.service symlink:

* I guess a preset might work, but this requires a post-install script which checks if we are upgrading from a version which had the symlink packaged to a version which has mooved to the preset and which then applies the preset. Without this on update the symlink is simply removed breaking the grub2 integration with "systemctl reboot --boot-loader-menu"

* It is rather unusual for a reboot.target service. We simply want this to always be enabled. And since bug 2230575 was only filed because of issues with the grub-boot-success.timer I believe the best fix here is to just restore the symlink, avoiding the need for a complex version checking post-install script

2. For the /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service symlink

Using a hardcoded symlink is the systemd upstream recommended way to integrate with systemd's offline updates, see:
https://www.freedesktop.org/software/systemd/man/latest/systemd.offline-updates.html#Recommendations
So lets just follow the actually recommended practice here and switch back to using a symlink inside the grub2-tools package *as recommended*

3. For the /usr/lib/systemd/user/timers.target.wants/grub-boot-success.timer

I'm not sure what to do here. Do presets even work for *user* units ?

@cglombek for now I believe we should revert your changes to fix bug 2230575 since they are causing all kind of issues. Can you please submit a pull-request to rever the changes ?

Also for future changes touching the grub2 <-> systemd integration / grub2's hidden menu support : https://hansdegoede.dreamwidth.org/19180.html please Cc me on the changes.

To be clear when I wrote "It is rather unusual for a reboot.target service" with "It" I meant using a preset for a reboot.target service is rather unusual.

I have just done a fresh F39 install and the presets do work in that case and everything works as it should.

Still I believe that for the grub2-systemd-integration.service grub-boot-indeterminate.service cases reverting the changes and switching back to a static packaged symlink is best. For system-update.target.wants using a static symlink is the recommended way and for reboot.target.wants grub now is the only package using an install section + preset.

I have been thinking about what to do about the grub-boot-success.timer on updates. If we can find a way to detect that we are coming from a package version older then the version with the to-be-written fix for this and then apply the preset once on updating from such an older version to that version (or newer) that would be best. I have been looking for a %post scriptlet recipe to do something like this but I have failed to find one and I'm afraid that such an approach will be fragile.

Alternatively I believe that the simplest and most robust solution is to add the /usr/lib/systemd/user/timers.target.wants/grub-boot-success.timer symlink to fedora-release-identity-workstation with a comment in the .spec file that "This historically was a static symlink and needs to stay one to not break update paths" (or something similar).

It looks like most of the desktop spins all have their own fedora-release-identity-* sub package, so the symlink will need to be added to the %files for all of the desktop fedora-release-identity-* sub packages, e.g. to at least all off:

fedora-release-identity-budgie
fedora-release-identity-cinnamon
fedora-release-identity-kde
fedora-release-identity-kinoite
fedora-release-identity-lxqt
fedora-release-identity-matecompiz
fedora-release-identity-mobility
fedora-release-identity-silverblue
fedora-release-identity-workstation
fedora-release-identity-xfce

I guess an even more KISS approach to the grub-boot-success.timer would be to add a %post which applies the preset unconditionally to fix the upgrade path, with a comment that this can be dropped after Fedora 42 .

Downside is that for users with a preset which enables it who still don't want it for some reason, this will cause the timer to get re-enabled on each grub package update. OTOH having the static symlink makes it impossible to disable it in a clean way and directly deleting the symlink has the same issue of it returning on every update of the package owning the link.

Yes thinking more about this I believe the right mix of fixes here is:

1. For grub2-systemd-integration.service grub-boot-indeterminate.service re-add the static symlink and drop the install sections and their presence in /usr/lib/systemd/system-preset/90-default.preset

2. For grub-boot-success.timer at a %post which applies the preset on every update (for now, to be removed in Fedora 42)

I've opened the following PRs for (1):

- https://github.com/rhboot/grub2/pull/142
- https://src.fedoraproject.org/rpms/fedora-release/pull-request/293
- https://src.fedoraproject.org/rpms/grub2/pull-request/40 (needs another update 

I don't have cycles currently to look into (2)

> I've opened the following PRs for (1):

Thank you.

> https://src.fedoraproject.org/rpms/grub2/pull-request/40

This does not build because you did not re-add the service symlinks to %files. Next time please do a local "fedpkg mockbuild" to ensure things build before submitting a pull-request and preferably also install the locally build packages and check that the intended changes are present.

I'm preparing a new src.fedoraproject.org/rpms/grub2/pull-request now which also addresses (2), so no need to fix this.

> I don't have cycles currently to look into (2)

TBH I'm not really happy with this. I'm a firm believer in you broke it you fix it and many of us have bandwidth issues.

Anyways this morning I managed to remember how to write scriptlets (triggers) which only run when upgrading from a version which is older then a specific version (the first version with the trigger). So we can fix (2) in a clean way. I plan to submit a pull-req for this soon.

I have submitted a new pull-request fully resolving this bug:

https://src.fedoraproject.org/rpms/grub2/pull-request/41

This resolves (1) by making the symlinks for the services static again and it resolves (2) by bumping grub2's version to 1:2.06-107 (std release bump) and adds a %triggerpostun which applies the grub-boot-success.timer preset when upgrading from versions older then 1:2.06-107.

I have tested both upgrading from version 2.06-98 where both services and user symlinks were still static as well as upgrading from version 2.06-104 where both services and user symlinks were relying on presets to enable them.

FEDORA-2023-4d409dcff3 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-4d409dcff3

FEDORA-2023-2a9508cf8d has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-2a9508cf8d

Nicolas, thank you for getting the fixes for this build and on their way to updates-testing.

FEDORA-2023-2a9508cf8d has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2a9508cf8d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2a9508cf8d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-4d409dcff3 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-4d409dcff3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-4d409dcff3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-4d409dcff3 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-2a9508cf8d has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.