Bug 1868834

Summary: EFI files sometimes corrupted after upgrade and reboot
Product: [Fedora] Fedora Reporter: Tom Anderson <twic>
Component: grub2Assignee: Javier Martinez Canillas <fmartine>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: fmartine, information, knordback, lkundrak, netllama, pjones
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-25 16:38:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tom Anderson 2020-08-13 23:12:55 UTC
Description of problem:

Sometimes, after i do a dnf update and then reboot, boot fails, i think because certain EFI files are corrupt.

I have raised this bug against grub2, but i don't know if that is the appropriate component - if not, please let me know. This could be a problem with dnf, hfsplus, rhboot, grub2, or user error.

Version-Release number of selected component (if applicable): Name: grub2-efi-x64 Version: 2.04 Release: 21.fc32

How reproducible: it is impossible to recreate the exact conditions which lead to the bug, so i can't answer this

Steps to Reproduce:
1. dnf update, and notice that there is a kernel upgrade
2. reboot

Actual results:

Boot fails, with this message on the screen:

Invalid image
Failed to read header: Unsupported
Failed to load header: Unsupported
start_image() returned Unsupported

Expected results:

Boot succeeds.

Additional info:

I am running Fedora 32 on a MacBook Pro "Core i7" 2.2 15" Early 2011, with an aftermarket 256GB Samsung 850 PRO SSD. Long ago, this machine ran OS X, but has been running Fedora for years now.

The /boot/efi partition is:

/dev/sda1 on /boot/efi type hfsplus (rw,relatime,umask=22,uid=0,gid=0,nls=utf8) [Linux HFS+ ESP]

Here is a summary of the steps i took to diagnose and fix the problem today:

https://gist.github.com/tomwhoiscontrary/a933ecc5b1b3a4dc399bd3a001b9c02b

Briefly:

0. boot from a live USB stick

1. fsck the machine's boot partition, finding a few small problems

2. find files which differ between the machine's boot partition and the live USB's boot partition

3. where any files differ, replace the machine's version with the live's USB version

4. reboot

The smoking gun is this checksum of a handful of EFI files from the live USB (/boot/efi) and the machine (/root/efi):

[root@localhost-live ~]# diff -rq /boot/efi /root/efi | grep differ | tr ' ' '\n' | grep /efi/ | sort -u | xargs cksum 
2669304285 1583432 /boot/efi/EFI/fedora/gcdia32.efi
2381972754 2500936 /boot/efi/EFI/fedora/gcdx64.efi
2530681163 1583432 /boot/efi/EFI/fedora/grubia32.efi
1608039898 2500936 /boot/efi/EFI/fedora/grubx64.efi
1226003682 1587528 /root/efi/EFI/fedora/gcdia32.efi
4294967295 0 /root/efi/EFI/fedora/gcdx64.efi
4294967295 0 /root/efi/EFI/fedora/grubia32.efi
4294967295 0 /root/efi/EFI/fedora/grubx64.efi

The files on the machine all appear to me to be truncated.

I had the same problem two months before; i analysed it slightly differently, but also saw what i interpreted as truncation of one of these files:

[root@localhost-live ~]# for filename in gcdia32.efi gcdx64.efi grubia32.efi grubx64.efi; do ls -l /boot/efi/EFI/fedora/${filename}; done
-rwx------. 1 root root 1583432 Apr  2 11:40 /boot/efi/EFI/fedora/gcdia32.efi
-rwx------. 1 root root 2500936 Apr  2 11:40 /boot/efi/EFI/fedora/gcdx64.efi
-rwx------. 1 root root 1583432 Apr  2 11:40 /boot/efi/EFI/fedora/grubia32.efi
-rwx------. 1 root root 2500936 Apr  2 11:40 /boot/efi/EFI/fedora/grubx64.efi
[root@localhost-live ~]# for filename in gcdia32.efi gcdx64.efi grubia32.efi grubx64.efi; do ls -l /root/efi/EFI/fedora/${filename}; done
-rwx------. 1 root root 1587528 May 19 14:10 /root/efi/EFI/fedora/gcdia32.efi
-rwx------. 1 root root 2513224 May 19 14:10 /root/efi/EFI/fedora/gcdx64.efi
-rwx------. 1 root root 1587528 May 19 14:10 /root/efi/EFI/fedora/grubia32.efi
--w-------. 2 root root 0 May 24 07:03 /root/efi/EFI/fedora/grubx64.efi

I know very little about these files. Perhaps this apparent truncation is actually an intentional part of an upgrade. If so, i suppose it doesn't work on my machine.

I note that my kludge here leaves the machine with out-of-date EFI files. I would be interested to know if there is a way to ensure they are up-to-date.

Comment 1 Tom Anderson 2020-08-16 17:19:18 UTC
I thought i'd take a closer look at the state of the files in /boot/efi/EFI/fedora:

[root@olympus ~]# find /boot/efi/EFI/fedora -type f | xargs rpm -qf | grep -v 'is not owned by any package' | sort -u | xargs rpm -V | sort -u
.M.......  c /boot/efi/EFI/fedora/grub.cfg
.M.......  c /boot/efi/EFI/fedora/grubenv
SM5....T.    /boot/efi/EFI/fedora/gcdia32.efi
SM5....T.    /boot/efi/EFI/fedora/gcdx64.efi
SM5....T.    /boot/efi/EFI/fedora/grubia32.efi
SM5....T.    /boot/efi/EFI/fedora/grubx64.efi

I've fixed the permissions, but that leaves the size, digest, and modification time different for the EFI files.

From the package names, it seem like some of these files are only needed on IA32, and some are only needed when booting off removable media. I'm not sure i need those at all.

I had a look at which packages were upgraded as part of the update which preceded the problem (which was update 208 in dnf history):

[root@olympus ~]# dnf history info 208 | sed -rn 's/^\s*(Install|Upgrade)\s+(\S+).*$/\2/p' | sort -u

Nothing in the lists stands out as being related to booting, apart from a kernel update.

Comment 2 Kurt Nordback 2020-11-26 01:06:21 UTC
For the record, I had the same problem. In my case it was on Mac Mini hardware from 2012, so Apple hardware is a common thread. Mine is dual-booted with Mac OS/X and FC (currently 32).

I did a

dnf upgrade --refresh

and rebooted, and got the same errors shown above:

Invalid image
Failed to read header: Unsupported
Failed to load header: Unsupported
start_image() returned Unsupported

Based on the suggestions here (thanks for those), I also found that my /boot/efi/EFI/fedora/mmx64.efi and /boot/efi/EFI/fedora/grubx64.efi were of size 0. I copied these from an FC33 live USB, along with gcdx64.efi, which didn't exist at all (but I have no idea if it's necessary). 

After that I can boot successfully into FC. 

Looking at my dnf history info, I do see that the upgrade affected some grub2 packages, including grub2-common and grub2-efi-x64.

Comment 3 Fedora Program Management 2021-04-29 16:35:42 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 4 Fedora Admin user for bugzilla script actions 2021-05-07 00:35:40 UTC
This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.

Comment 5 Ben Cotton 2021-05-25 16:38:41 UTC
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.