Bug 1958540

Could you attach the following files:

/boot/efi/EFI/fedora/grub.cfg
/boot/grub2/grub.cfg
/boot/efi/EFI/fedora/grubenv
/boot/grub2/grubenv
/etc/default/grub

And post the result from
ls -l /boot/loader/entries
efibootmgr -v

With the SSD temporarily connected to another computer, I can now copy the requested files.

# ls -l /run/media/beorn/speedy.boot2/loader/entries/
totalt 16
-rw-r--r--. 1 root root 383 15 maj  2020 69d27b356a94476da859461d3a3bc6fd-0-rescue.conf
-rw-r--r--. 1 root root 306 28 apr 16.56 69d27b356a94476da859461d3a3bc6fd-5.11.17-100.fc32.x86_64.conf
-rw-r--r--. 1 root root 320  8 maj 01.14 69d27b356a94476da859461d3a3bc6fd-5.11.17-300.fc34.x86_64.conf
-rw-r--r--. 1 root root 301 17 mar 20.38 69d27b356a94476da859461d3a3bc6fd-5.11.7-100.fc32.x86_64.conf

I don't think I can use efibootmgr until I get the system running. I tried passing --disk to make it look at the temporarily connected SSD, but the output still contains the UUID of this other computer's EFI partition.

Created attachment 1781164 [details]
/boot/efi/EFI/fedora/grub.cfg

Created attachment 1781165 [details]
/boot/grub2/grub.cfg

Created attachment 1781166 [details]
/boot/efi/EFI/fedora/grubenv.rpmsave

There is no /boot/efi/EFI/fedora/grubenv, only a grubenv.rpmsave.

Created attachment 1781167 [details]
/boot/grub2/grubenv

Created attachment 1781168 [details]
/etc/default/grub

The /boot/grub2/grub.cfg looks stale and also pre-BLS. Somehow it wasn't replaced by /boot/efi/EFI/fedora/grub.cfg when the grub2-common script ran?

Has 'grub2-install' ever been used on this system? Another possibility is that grub2-install was using /boot/grub2/grub.cfg which left /boot/efi/EFI/fedora/grub.cfg to go unused and stale. Upon F34 upgrade, then stale /boot/efi/EFI/fedora/grub.cfg would have replaced /boot/grub2/grub.cfg, thus breaking boot. Hmm.

Suggestion for fixing it:
1a. Either use a Fedora netinstaller to boot, using the Troubleshooting->Rescue a Fedora system option in the GRUB menu; this assembles things for you; OR
1b. Manually assemble it. There's a hint how to do it in the storage.log from the original installation:

grep -E 'mount.*sysimage' /var/log/anaconda/storage.log

2. chroot /mnt/sysimage
3. grub2-mkconfig -o /boot/grub2/grub.cfg   #same on UEFI as on BIOS now
4. dnf reinstall shim-efi-* grub2-efi-*   ##mostly to make sure you've got the proper grubx64.efi in case grub2-install was previously used
5. exit; reboot

efibootmgr -v should have an entry pointing to shimx64.efi on the EFI system partition

Maybe 'stat /boot/grub2/grub.cfg' before you replace it with grub2-mkconfig? That might tell us if it's been there a while and thus wasn't replaced; or if it was moved from the ESP.

> Has 'grub2-install' ever been used on this system?

I have needed to fix boot-related things before. I don't remember exactly what commands I used years ago.

> Another possibility is that grub2-install was using /boot/grub2/grub.cfg which left /boot/efi/EFI/fedora/grub.cfg to go unused and stale. Upon F34 upgrade, then stale /boot/efi/EFI/fedora/grub.cfg would have replaced /boot/grub2/grub.cfg, thus breaking boot.

Could be, if I followed some instructions that weren't quite optimal. I've often been confused about what was the proper place for grub-related files.

Note, though, that installing kernel updates and rebooting to the new kernel has always worked until now. If one grub.cfg was in use and the other was stale, then the code that updated the boot menu with new kernels must have known which file to write. Therefore it should have been possible to know the same during the F34 upgrade, and avoid replacing the file in use with the stale one.

> Maybe 'stat /boot/grub2/grub.cfg' before you replace it with grub2-mkconfig?

# LANG=C stat /run/media/beorn/speedy.boot2/grub2/grub.cfg
  File: /run/media/beorn/speedy.boot2/grub2/grub.cfg
  Size: 12467     	Blocks: 32         IO Block: 4096   regular file
Device: 881h/2177d	Inode: 18          Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:boot_t:s0
Access: 2021-05-08 21:50:39.132704644 +0200
Modify: 2020-05-15 19:37:39.202821977 +0200
Change: 2021-05-08 01:14:32.600006844 +0200
 Birth: -

Apparently the inode was created or modified during the upgrade, while the file content is older, so yes, it looks like it was moved.

I'll just fyi this:

https://bugzilla.redhat.com/show_bug.cgi?id=1625124#c54

Not exactly the same scenario, but might shed some light on fixes :-/

_Since_ fixing after _my_ failed grub-menu escapades, subsequent upgrades have gone smoothly, fwiw.

I also see in my (old) notes some cleanup I did to straighten out the then (failed) switch to BLS,

dnf -y remove grubby-deprecated
rm -f /boot/loader/entries/*
rm -f /boot/*rescue*
LIST=$( ls -1 /boot/*vmli* | sed -e 's|/boot/vmlinuz-||g' -e 's|*$||g')
for k in $LIST
do
 /bin/kernel-install add $k /lib/modules/${k}/vmlinuz
done
ls -al /boot/loader/entries/
	...

Where you able to fix this issue? You could do the following
to get a working GRUB configuration (backup your files first
if you want to be sure):

$ rm /boot/efi/EFI/fedora/grub.cfg
$ rm /boot/grub2/grub.cfg

$ sudo dnf reinstall grub2-common

Just double check if the files are correct before rebooting.

Fixing the issue would mean to change some program so that grub.cfg would not be replaced with a stale version if the circumstances would be reproduced. In that sense the issue has not been fixed as far as I know.

I worked around the problem in my own case by replacing /boot/grub2/grub.cfg with the following:

insmod part_gpt
insmod ext2
search --no-floppy --fs-label --set=root speedy.boot2
insmod blscfg
blscfg

That was good enough to find the kernel and get Fedora going. (Keeping all the partitions labeled pays off in situations like this.) Then I ran "grub2-mkconfig -o /boot/grub2/grub.cfg" to get the boot working normally.

Is unclear what the problem is and I was not able to reproduce this. The user has already worked around
the breakage so I'll just close this with insufficient data.

I've tried step in comment 12. The error is still here.
Nb. /etc/default/grub has this line:

GRUB_THEME=/boot/grub2/themes/system/theme.txt

and


$ rpm -qf /boot/grub2/themes/system/theme.txt
grub2-starfield-theme-2.02-0.40.fc26.x86_64

Is there another package providing system/theme.txt in Fedora?

I think there's not enough information to reproduce the original conditions that resulted in upgrade confusion. There is now updated guidance in the wiki for reinstalling GRUB2. 

https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Reinstalling_GRUB_2

If that doesn't help, file a new bug against product "Fedora Documentation" and reference this bug and the GRUB2 wiki, and cc me on it.

Fedora hasn't included themes with any Fedora edition or spin in a very long time. It would take some research to see if it's possible to support themes, but until then it's up to users to self-support. Upstream has a user support list that might be able to help.