Hi, after some recent change, https://fedoraproject.org/wiki/Changes/HiddenGrubMenu is now broken after performing an offline system update. After performing a system update, then on the next boot the GRUB boot menu allowing kernel selection will improperly appear. This is only supposed to happen if the previous boot was unsuccessful. I asked Hans and he says that offline updates are supposed to set the boot_indeterminate GRUB environment counter instead of the boot_success counter. I assume something is going wrong somewhere when either setting or reading those counters. I think it broke sometime in the past few months. I first noticed it after upgrading to Fedora 39, but it actually also affects Fedora 38 too. Reproducible: Always
Hans coincidentally just hit the issue himself and says the problem is caused by boot_indeterminate not being set in grubenv during the offline update. He will investigate.
The issue seems to be that grub2-tools no longer ships the /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service symlink: Working F38 system: [hans@shalem ~]$ rpm -q grub2-tools grub2-tools-2.06-95.fc38.x86_64 [hans@shalem ~]$ rpm -ql grub2-tools | grep grub-boot-indeterminate.service /usr/lib/systemd/system/grub-boot-indeterminate.service /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service Broken F39 system: [root@fedora ~]# rpm -q grub2-tools grub2-tools-2.06-104.fc39.x86_64 [root@fedora ~]# rpm -ql grub2-tools | grep grub-boot-indeterminate.service /usr/lib/systemd/system/grub-boot-indeterminate.service Note after installing recent updates on F38 it breaks there too: [hans@shalem ipu6-drivers]$ rpm -q grub2-tools grub2-tools-2.06-102.fc38.x86_64 [hans@shalem ipu6-drivers]$ rpm -ql grub2-tools | grep grub-boot-indeterminate.service /usr/lib/systemd/system/grub-boot-indeterminate.service Workaround (untested): sudo ln -s ../grub-boot-indeterminate.service /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service Fixing this should be a matter of just re-adding the accidentally dropped symlink to grub2-tools.
Asking the boot menu to be shown by e.g. keeping alt pressed when hitting the "restart" button on the restart? cancel/restart dialog in GNOME also no longer works. The root cause is the same, in this case it is the: /usr/lib/systemd/system/reboot.target.wants/grub2-systemd-integration.service symlink which is missing in later grub builds. Workaround: sudo ln -s ../grub2-systemd-integration.service /usr/lib/systemd/system/reboot.target.wants/grub2-systemd-integration.service
And I just noticed a third issue due to another missing .wants symlink, sessions which last longer then 2 minutes used to count as boot-success, but they no longer do since the grub-boot-success.timer is no longer enabled by default. Workaround: sudo mkdir /usr/lib/systemd/user/timers.target.wants sudo ln -s ../grub-boot-success.timer /usr/lib/systemd/user/timers.target.wants/grub-boot-success.timer Looking at grub's package history this seems to be caused by the fixes for bug 2230575, which removed all 3 .wants symlinks. This is wrong for a variety of reasons: 1. For the /usr/lib/systemd/system/reboot.target.wants/grub2-systemd-integration.service symlink: * I guess a preset might work, but this requires a post-install script which checks if we are upgrading from a version which had the symlink packaged to a version which has mooved to the preset and which then applies the preset. Without this on update the symlink is simply removed breaking the grub2 integration with "systemctl reboot --boot-loader-menu" * It is rather unusual for a reboot.target service. We simply want this to always be enabled. And since bug 2230575 was only filed because of issues with the grub-boot-success.timer I believe the best fix here is to just restore the symlink, avoiding the need for a complex version checking post-install script 2. For the /usr/lib/systemd/system/system-update.target.wants/grub-boot-indeterminate.service symlink Using a hardcoded symlink is the systemd upstream recommended way to integrate with systemd's offline updates, see: https://www.freedesktop.org/software/systemd/man/latest/systemd.offline-updates.html#Recommendations So lets just follow the actually recommended practice here and switch back to using a symlink inside the grub2-tools package *as recommended* 3. For the /usr/lib/systemd/user/timers.target.wants/grub-boot-success.timer I'm not sure what to do here. Do presets even work for *user* units ? @cglombek for now I believe we should revert your changes to fix bug 2230575 since they are causing all kind of issues. Can you please submit a pull-request to rever the changes ? Also for future changes touching the grub2 <-> systemd integration / grub2's hidden menu support : https://hansdegoede.dreamwidth.org/19180.html please Cc me on the changes.
To be clear when I wrote "It is rather unusual for a reboot.target service" with "It" I meant using a preset for a reboot.target service is rather unusual.
I have just done a fresh F39 install and the presets do work in that case and everything works as it should. Still I believe that for the grub2-systemd-integration.service grub-boot-indeterminate.service cases reverting the changes and switching back to a static packaged symlink is best. For system-update.target.wants using a static symlink is the recommended way and for reboot.target.wants grub now is the only package using an install section + preset. I have been thinking about what to do about the grub-boot-success.timer on updates. If we can find a way to detect that we are coming from a package version older then the version with the to-be-written fix for this and then apply the preset once on updating from such an older version to that version (or newer) that would be best. I have been looking for a %post scriptlet recipe to do something like this but I have failed to find one and I'm afraid that such an approach will be fragile. Alternatively I believe that the simplest and most robust solution is to add the /usr/lib/systemd/user/timers.target.wants/grub-boot-success.timer symlink to fedora-release-identity-workstation with a comment in the .spec file that "This historically was a static symlink and needs to stay one to not break update paths" (or something similar). It looks like most of the desktop spins all have their own fedora-release-identity-* sub package, so the symlink will need to be added to the %files for all of the desktop fedora-release-identity-* sub packages, e.g. to at least all off: fedora-release-identity-budgie fedora-release-identity-cinnamon fedora-release-identity-kde fedora-release-identity-kinoite fedora-release-identity-lxqt fedora-release-identity-matecompiz fedora-release-identity-mobility fedora-release-identity-silverblue fedora-release-identity-workstation fedora-release-identity-xfce
I guess an even more KISS approach to the grub-boot-success.timer would be to add a %post which applies the preset unconditionally to fix the upgrade path, with a comment that this can be dropped after Fedora 42 . Downside is that for users with a preset which enables it who still don't want it for some reason, this will cause the timer to get re-enabled on each grub package update. OTOH having the static symlink makes it impossible to disable it in a clean way and directly deleting the symlink has the same issue of it returning on every update of the package owning the link. Yes thinking more about this I believe the right mix of fixes here is: 1. For grub2-systemd-integration.service grub-boot-indeterminate.service re-add the static symlink and drop the install sections and their presence in /usr/lib/systemd/system-preset/90-default.preset 2. For grub-boot-success.timer at a %post which applies the preset on every update (for now, to be removed in Fedora 42)
I've opened the following PRs for (1): - https://github.com/rhboot/grub2/pull/142 - https://src.fedoraproject.org/rpms/fedora-release/pull-request/293 - https://src.fedoraproject.org/rpms/grub2/pull-request/40 (needs another update I don't have cycles currently to look into (2)
> I've opened the following PRs for (1): Thank you. > https://src.fedoraproject.org/rpms/grub2/pull-request/40 This does not build because you did not re-add the service symlinks to %files. Next time please do a local "fedpkg mockbuild" to ensure things build before submitting a pull-request and preferably also install the locally build packages and check that the intended changes are present. I'm preparing a new src.fedoraproject.org/rpms/grub2/pull-request now which also addresses (2), so no need to fix this. > I don't have cycles currently to look into (2) TBH I'm not really happy with this. I'm a firm believer in you broke it you fix it and many of us have bandwidth issues. Anyways this morning I managed to remember how to write scriptlets (triggers) which only run when upgrading from a version which is older then a specific version (the first version with the trigger). So we can fix (2) in a clean way. I plan to submit a pull-req for this soon.
I have submitted a new pull-request fully resolving this bug: https://src.fedoraproject.org/rpms/grub2/pull-request/41 This resolves (1) by making the symlinks for the services static again and it resolves (2) by bumping grub2's version to 1:2.06-107 (std release bump) and adds a %triggerpostun which applies the grub-boot-success.timer preset when upgrading from versions older then 1:2.06-107. I have tested both upgrading from version 2.06-98 where both services and user symlinks were still static as well as upgrading from version 2.06-104 where both services and user symlinks were relying on presets to enable them.
FEDORA-2023-4d409dcff3 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-4d409dcff3
FEDORA-2023-2a9508cf8d has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-2a9508cf8d
Nicolas, thank you for getting the fixes for this build and on their way to updates-testing.
FEDORA-2023-2a9508cf8d has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2a9508cf8d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2a9508cf8d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-4d409dcff3 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-4d409dcff3` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-4d409dcff3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-4d409dcff3 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-2a9508cf8d has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.