Bug 1870035

Summary: The "grubx64.efi" provided by latest grub2-efi-x64-2.02-0.86.el7_8.x86_64 package have broken PXE based discovery for EFI systems on Satellite 6.7
Product: Red Hat Satellite Reporter: Sayan Das <saydas>
Component: Discovery ImageAssignee: Lukas Zapletal <lzap>
Status: CLOSED ERRATA QA Contact: Roman Plevka <rplevka>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: djuarezg, fmartine, javierm, lzap, michael.mcconachie, rabajaj, zhunting
Target Milestone: 6.9.0Keywords: Regression, Triaged
Target Release: UnusedFlags: lukas.crockett: needinfo-
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: foreman-2.3.1.14-1 Doc Type: Known Issue
Doc Text:
Cause: A security update in RHEL 7.8 in grub package (grub2-2.02-0.86.el7) caused PXE to timeout when downloading larger initramdisks, foreman discovery image for example. RHEL team is tracking the bug as https://bugzilla.redhat.com/show_bug.cgi?id=1869987 Consequence: Discovery of nodes is not possible. Workaround (if any): Downgrade to version grub2-2.02-0.85.el7 or upgrade to version grub2-2.02-0.87.el7 from RHEL 7.9 release.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-21 13:17:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
First error
none
Second error none

Description Sayan Das 2020-08-19 08:50:32 UTC 
Description of problem:

Issue started after OS + Satellite was patched and upgraded to Satellite 6.7.2.

The pxe-based discovery process fails to load the fdi initrd image on any efi based system and hence fails to boot into discovery mode. This happens only when the "grubx64.efi" provided by latest grub2-efi-x64-2.02-0.86.el7_8.x86_64 package is present on the Satellite 6.7.


Version-Release number of selected component (if applicable):

Only first three are main concerns,

* satellite 6.7
* grub2-efi-x64-2.02-0.86.el7_8.x86_64  [ md5sum: 31016d39e449db29b3d8ca4eb18b14e0  /var/lib/tftpboot/grub2/grubx64.efi ]
* foreman-discovery-image-3.5.4-8.el7sat.noarch
* foreman-bootloaders-redhat-201901011200-1.el7sat.noarch
* foreman-bootloaders-redhat-tftpboot-201901011200-1.el7sat.noarch



How reproducible:
100%


Steps to Reproduce:
1. On any minor release of Satellite 6.7, download the "grub2-efi-x64-2.02-0.86.el7_8.x86_64" and extract it.
2. Replace the existing "/var/lib/tftpboot/grub2/grubx64.efi" with the "grubx64.efi" provided by above package and ensure correct permission\ownership\selinux_label is set.
3. Attempt to create a EFI based VM and boot it in the build network of satellite to build it via PXE based discovery process.
4. Select the option "Foreman Discovery Image - efi" to Discover and further build the system



Actual results:

* System will hang for a while and then will show --> error: timeout reading `boot/foreman-discovery-image-3.5.4-8.iso-img`
* It will continue to boot with loading the actual "boot/fdi-image/initrd0.img" system, will fail to mount the squashfs and the drop to dracut shell.
* Screenshots and rdsosreport from test environments will be attached shortly.
* Use a "grubx64.efi" from older "grub2-efi-x64" and the discovery works just fine.

Expected results:
* System should be getting discovered and build properly.


Additional info:
* This only affects the EFI based discovery
* Under the same scenario, if we create a EFI based system with simple network-based deployment that will work just fine.
Comment 1 Sayan Das 2020-08-19 08:52:42 UTC 
Created attachment 1711823 [details]
First error
Comment 2 Sayan Das 2020-08-19 08:53:14 UTC 
Created attachment 1711824 [details]
Second error
Comment 12 Lukas Zapletal 2020-08-24 10:57:45 UTC 
Workaround: Downgrade grub2 and run installer or extract and replace /var/lib/tftpboot/grub2/grubx64.efi file from grub2-efi-x64 RPM package. For quick download here are all builds (RPM, EFI files) so you can test with any of the versions mentioned above (x86_64):

http://people.redhat.com/~lzapleta/grub/
Comment 15 Lukas Zapletal 2020-08-24 14:19:18 UTC 
WORKAROUND WITH HOTFIX:

http://people.redhat.com/~lzapleta/grub/grub2-efi-x64-cdboot-2.02-0.87.el7.hotfix/grubx64.efi

SecureBoot will not work as this file is not signed.
Comment 21 michael.mcconachie 2020-10-22 20:54:37 UTC 
@lzap - backporting /var/lib/tftpboot/grub2/grubx64.efi from the file in the RPM you shared fixed our CentOS7 Foreman 2.14 instance that was also unable to make use of the fdi image on UEFI enabled hosts. THANK YOU!
Comment 22 Lukas Zapletal 2020-10-23 07:11:24 UTC 
Nice, I am assuming you don't have any additional request and the NEEDINFO flag was a mistake. Thanks!

For the record, you can't go wrong with Rawhide version: https://archives.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/x86_64/os/EFI/BOOT/grubx64.efi
Comment 24 Roman Plevka 2021-02-23 14:08:38 UTC 
FAILQA
for 6.9.0-14.0

using grub2-efi-x64-2.02-0.87.el7.x86_64
GRUB seems to be unable to process the default grub.cfg file.

using a default workflow with booting an unknown host, i can see the EFI requesting and retrieving grubx64.efi and grub.cfg but it falls back to the grub shell.
Comment 25 Lukas Zapletal 2021-02-23 14:32:14 UTC 
We are hit by a different but, we commited a change that added

echo blah blah can't blah blah

which obviously breaks grub2 parser and it goes into a black screen. Ouch.

Please cherry pick this trivial fix: https://github.com/theforeman/foreman/pull/8317/files

Thanks
Comment 27 Javier Martinez Canillas 2021-02-23 14:36:34 UTC 
(In reply to Roman Plevka from comment #24)
> FAILQA
> for 6.9.0-14.0
> 
> using grub2-efi-x64-2.02-0.87.el7.x86_64

Did you try with the grub2-efi-x64-cdboot-2.02-0.87.el7.x86_64 package (note the
-cdboot suffix before the package NVR), this sets GRUB prefix to /EFI/BOOT instead
of the /EFI/redhat/ path.

> GRUB seems to be unable to process the default grub.cfg file.
> 
> using a default workflow with booting an unknown host, i can see the EFI
> requesting and retrieving grubx64.efi and grub.cfg but it falls back to the
> grub shell.

What's the path of grub.cfg requested ?
Comment 28 Roman Plevka 2021-02-23 14:45:00 UTC 
(In reply to Lukas Zapletal from comment #25)
> We are hit by a different but, we commited a change that added
> 
> echo blah blah can't blah blah
> 
> which obviously breaks grub2 parser and it goes into a black screen. Ouch.
> 
> Please cherry pick this trivial fix:
> https://github.com/theforeman/foreman/pull/8317/files
> 
> Thanks

I can confirm that https://github.com/theforeman/foreman/pull/8299 fixes the issue.
Comment 30 Roman Plevka 2021-03-10 13:56:58 UTC 
VERIFIED
on sat6.9.0-16.0

grub config is now valid and is loaded properly.
Comment 33 errata-xmlrpc 2021-04-21 13:17:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313