Bug 1958540 - Upgrade to Fedora 34 broke the boot menu.
Summary: Upgrade to Fedora 34 broke the boot menu.
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: 34
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Javier Martinez Canillas
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-08 16:31 UTC by Björn Persson
Modified: 2021-07-09 05:48 UTC (History)
6 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2021-07-09 05:48:32 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
/boot/efi/EFI/fedora/grub.cfg (143 bytes, text/plain)
2021-05-08 21:25 UTC, Björn Persson
no flags Details
/boot/grub2/grub.cfg (12.17 KB, text/plain)
2021-05-08 21:28 UTC, Björn Persson
no flags Details
/boot/efi/EFI/fedora/grubenv.rpmsave (1.00 KB, text/plain)
2021-05-08 21:35 UTC, Björn Persson
no flags Details
/boot/grub2/grubenv (1.00 KB, text/plain)
2021-05-08 21:39 UTC, Björn Persson
no flags Details
/etc/default/grub (343 bytes, text/plain)
2021-05-08 21:43 UTC, Björn Persson
no flags Details

Description Björn Persson 2021-05-08 16:31:28 UTC
I used yum system-upgrade to upgrade from Fedora 32 to Fedora 34. Now Grub complains about not finding some theme files, and then displays a menu with two kernels from Fedora 29 and a rescue entry from Fedora 28. If I choose one of the Fedora 29 entries, then Grub unsurprisingly fails to find the kernel. The rescue entry actually boots to some kind of shell, after printing lots of timeout messages, but getting from there to a working system will be a major research project for me.

The menu also has entries for "advanced flags" (or something like that; not sure what it would be in English) and "tboot". Both lead to other menus with boot entries from Fedora 25.

The boot partition contains three sets of vmlinuz, initramfs, config and System.map files – one fc34 set and two fc32 sets as expected – but Grub has apparently reverted to a years-old boot menu program.

Comment 1 Chris Murphy 2021-05-08 18:38:38 UTC
Could you attach the following files:

/boot/efi/EFI/fedora/grub.cfg
/boot/grub2/grub.cfg
/boot/efi/EFI/fedora/grubenv
/boot/grub2/grubenv
/etc/default/grub

And post the result from
ls -l /boot/loader/entries
efibootmgr -v

Comment 2 Björn Persson 2021-05-08 21:24:00 UTC
With the SSD temporarily connected to another computer, I can now copy the requested files.

# ls -l /run/media/beorn/speedy.boot2/loader/entries/
totalt 16
-rw-r--r--. 1 root root 383 15 maj  2020 69d27b356a94476da859461d3a3bc6fd-0-rescue.conf
-rw-r--r--. 1 root root 306 28 apr 16.56 69d27b356a94476da859461d3a3bc6fd-5.11.17-100.fc32.x86_64.conf
-rw-r--r--. 1 root root 320  8 maj 01.14 69d27b356a94476da859461d3a3bc6fd-5.11.17-300.fc34.x86_64.conf
-rw-r--r--. 1 root root 301 17 mar 20.38 69d27b356a94476da859461d3a3bc6fd-5.11.7-100.fc32.x86_64.conf

I don't think I can use efibootmgr until I get the system running. I tried passing --disk to make it look at the temporarily connected SSD, but the output still contains the UUID of this other computer's EFI partition.

Comment 3 Björn Persson 2021-05-08 21:25:41 UTC
Created attachment 1781164 [details]
/boot/efi/EFI/fedora/grub.cfg

Comment 4 Björn Persson 2021-05-08 21:28:54 UTC
Created attachment 1781165 [details]
/boot/grub2/grub.cfg

Comment 5 Björn Persson 2021-05-08 21:35:39 UTC
Created attachment 1781166 [details]
/boot/efi/EFI/fedora/grubenv.rpmsave

There is no /boot/efi/EFI/fedora/grubenv, only a grubenv.rpmsave.

Comment 6 Björn Persson 2021-05-08 21:39:48 UTC
Created attachment 1781167 [details]
/boot/grub2/grubenv

Comment 7 Björn Persson 2021-05-08 21:43:47 UTC
Created attachment 1781168 [details]
/etc/default/grub

Comment 8 Chris Murphy 2021-05-08 22:12:50 UTC
The /boot/grub2/grub.cfg looks stale and also pre-BLS. Somehow it wasn't replaced by /boot/efi/EFI/fedora/grub.cfg when the grub2-common script ran?

Has 'grub2-install' ever been used on this system? Another possibility is that grub2-install was using /boot/grub2/grub.cfg which left /boot/efi/EFI/fedora/grub.cfg to go unused and stale. Upon F34 upgrade, then stale /boot/efi/EFI/fedora/grub.cfg would have replaced /boot/grub2/grub.cfg, thus breaking boot. Hmm.

Suggestion for fixing it:
1a. Either use a Fedora netinstaller to boot, using the Troubleshooting->Rescue a Fedora system option in the GRUB menu; this assembles things for you; OR
1b. Manually assemble it. There's a hint how to do it in the storage.log from the original installation:

grep -E 'mount.*sysimage' /var/log/anaconda/storage.log

2. chroot /mnt/sysimage
3. grub2-mkconfig -o /boot/grub2/grub.cfg   #same on UEFI as on BIOS now
4. dnf reinstall shim-efi-* grub2-efi-*   ##mostly to make sure you've got the proper grubx64.efi in case grub2-install was previously used
5. exit; reboot

efibootmgr -v should have an entry pointing to shimx64.efi on the EFI system partition

Comment 9 Chris Murphy 2021-05-08 22:28:36 UTC
Maybe 'stat /boot/grub2/grub.cfg' before you replace it with grub2-mkconfig? That might tell us if it's been there a while and thus wasn't replaced; or if it was moved from the ESP.

Comment 10 Björn Persson 2021-05-09 16:24:39 UTC
> Has 'grub2-install' ever been used on this system?

I have needed to fix boot-related things before. I don't remember exactly what commands I used years ago.

> Another possibility is that grub2-install was using /boot/grub2/grub.cfg which left /boot/efi/EFI/fedora/grub.cfg to go unused and stale. Upon F34 upgrade, then stale /boot/efi/EFI/fedora/grub.cfg would have replaced /boot/grub2/grub.cfg, thus breaking boot.

Could be, if I followed some instructions that weren't quite optimal. I've often been confused about what was the proper place for grub-related files.

Note, though, that installing kernel updates and rebooting to the new kernel has always worked until now. If one grub.cfg was in use and the other was stale, then the code that updated the boot menu with new kernels must have known which file to write. Therefore it should have been possible to know the same during the F34 upgrade, and avoid replacing the file in use with the stale one.

> Maybe 'stat /boot/grub2/grub.cfg' before you replace it with grub2-mkconfig?

# LANG=C stat /run/media/beorn/speedy.boot2/grub2/grub.cfg
  File: /run/media/beorn/speedy.boot2/grub2/grub.cfg
  Size: 12467     	Blocks: 32         IO Block: 4096   regular file
Device: 881h/2177d	Inode: 18          Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:boot_t:s0
Access: 2021-05-08 21:50:39.132704644 +0200
Modify: 2020-05-15 19:37:39.202821977 +0200
Change: 2021-05-08 01:14:32.600006844 +0200
 Birth: -

Apparently the inode was created or modified during the upgrade, while the file content is older, so yes, it looks like it was moved.

Comment 11 pgnet.dev 2021-05-11 16:49:41 UTC
I'll just fyi this:

https://bugzilla.redhat.com/show_bug.cgi?id=1625124#c54

Not exactly the same scenario, but might shed some light on fixes :-/

_Since_ fixing after _my_ failed grub-menu escapades, subsequent upgrades have gone smoothly, fwiw.

I also see in my (old) notes some cleanup I did to straighten out the then (failed) switch to BLS,

dnf -y remove grubby-deprecated
rm -f /boot/loader/entries/*
rm -f /boot/*rescue*
LIST=$( ls -1 /boot/*vmli* | sed -e 's|/boot/vmlinuz-||g' -e 's|*$||g')
for k in $LIST
do
 /bin/kernel-install add $k /lib/modules/${k}/vmlinuz
done
ls -al /boot/loader/entries/
	...

Comment 12 Javier Martinez Canillas 2021-06-22 15:07:00 UTC
Where you able to fix this issue? You could do the following
to get a working GRUB configuration (backup your files first
if you want to be sure):

$ rm /boot/efi/EFI/fedora/grub.cfg
$ rm /boot/grub2/grub.cfg

$ sudo dnf reinstall grub2-common

Just double check if the files are correct before rebooting.

Comment 13 Björn Persson 2021-06-22 18:07:17 UTC
Fixing the issue would mean to change some program so that grub.cfg would not be replaced with a stale version if the circumstances would be reproduced. In that sense the issue has not been fixed as far as I know.

I worked around the problem in my own case by replacing /boot/grub2/grub.cfg with the following:

insmod part_gpt
insmod ext2
search --no-floppy --fs-label --set=root speedy.boot2
insmod blscfg
blscfg

That was good enough to find the kernel and get Fedora going. (Keeping all the partitions labeled pays off in situations like this.) Then I ran "grub2-mkconfig -o /boot/grub2/grub.cfg" to get the boot working normally.

Comment 14 Javier Martinez Canillas 2021-06-29 17:06:48 UTC
Is unclear what the problem is and I was not able to reproduce this. The user has already worked around
the breakage so I'll just close this with insufficient data.

Comment 15 Tomasz Torcz 2021-07-08 18:28:55 UTC
I've tried step in comment 12. The error is still here.
Nb. /etc/default/grub has this line:

GRUB_THEME=/boot/grub2/themes/system/theme.txt

and


$ rpm -qf /boot/grub2/themes/system/theme.txt
grub2-starfield-theme-2.02-0.40.fc26.x86_64

Is there another package providing system/theme.txt in Fedora?

Comment 16 Chris Murphy 2021-07-09 05:48:32 UTC
I think there's not enough information to reproduce the original conditions that resulted in upgrade confusion. There is now updated guidance in the wiki for reinstalling GRUB2. 

https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Reinstalling_GRUB_2

If that doesn't help, file a new bug against product "Fedora Documentation" and reference this bug and the GRUB2 wiki, and cc me on it.

Fedora hasn't included themes with any Fedora edition or spin in a very long time. It would take some research to see if it's possible to support themes, but until then it's up to users to self-support. Upstream has a user support list that might be able to help.


Note You need to log in before you can comment on or make changes to this bug.