Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1870035 - The "grubx64.efi" provided by latest grub2-efi-x64-2.02-0.86.el7_8.x86_64 package have broken PXE based discovery for EFI systems on Satellite 6.7
Summary: The "grubx64.efi" provided by latest grub2-efi-x64-2.02-0.86.el7_8.x86_64 pac...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Discovery Image
Version: 6.7.0
Hardware: All
OS: Linux
unspecified
high
Target Milestone: 6.9.0
Assignee: Lukas Zapletal
QA Contact: Roman Plevka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-19 08:50 UTC by Sayan Das
Modified: 2023-12-15 18:55 UTC (History)
7 users (show)

Fixed In Version: foreman-2.3.1.14-1
Doc Type: Known Issue
Doc Text:
Cause: A security update in RHEL 7.8 in grub package (grub2-2.02-0.86.el7) caused PXE to timeout when downloading larger initramdisks, foreman discovery image for example. RHEL team is tracking the bug as https://bugzilla.redhat.com/show_bug.cgi?id=1869987 Consequence: Discovery of nodes is not possible. Workaround (if any): Downgrade to version grub2-2.02-0.85.el7 or upgrade to version grub2-2.02-0.87.el7 from RHEL 7.9 release.
Clone Of:
Environment:
Last Closed: 2021-04-21 13:17:39 UTC
Target Upstream Version:
Embargoed:
lukas.crockett: needinfo-

Attachments (Terms of Use)
First error (13.27 KB, image/png)
2020-08-19 08:52 UTC, Sayan Das 		no flags Details
Second error (74.84 KB, image/png)
2020-08-19 08:53 UTC, Sayan Das 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 31758 0 Normal Closed Unterminated single quote in chainloader pxeboot template breaks grub2 parser 2021-02-24 20:25:36 UTC
Red Hat Product Errata RHSA-2021:1313 0 None None None 2021-04-21 13:17:57 UTC

Description Sayan Das 2020-08-19 08:50:32 UTC 
Description of problem:

Issue started after OS + Satellite was patched and upgraded to Satellite 6.7.2.

The pxe-based discovery process fails to load the fdi initrd image on any efi based system and hence fails to boot into discovery mode. This happens only when the "grubx64.efi" provided by latest grub2-efi-x64-2.02-0.86.el7_8.x86_64 package is present on the Satellite 6.7.


Version-Release number of selected component (if applicable):

Only first three are main concerns,

* satellite 6.7
* grub2-efi-x64-2.02-0.86.el7_8.x86_64  [ md5sum: 31016d39e449db29b3d8ca4eb18b14e0  /var/lib/tftpboot/grub2/grubx64.efi ]
* foreman-discovery-image-3.5.4-8.el7sat.noarch
* foreman-bootloaders-redhat-201901011200-1.el7sat.noarch
* foreman-bootloaders-redhat-tftpboot-201901011200-1.el7sat.noarch



How reproducible:
100%


Steps to Reproduce:
1. On any minor release of Satellite 6.7, download the "grub2-efi-x64-2.02-0.86.el7_8.x86_64" and extract it.
2. Replace the existing "/var/lib/tftpboot/grub2/grubx64.efi" with the "grubx64.efi" provided by above package and ensure correct permission\ownership\selinux_label is set.
3. Attempt to create a EFI based VM and boot it in the build network of satellite to build it via PXE based discovery process.
4. Select the option "Foreman Discovery Image - efi" to Discover and further build the system



Actual results:

* System will hang for a while and then will show --> error: timeout reading `boot/foreman-discovery-image-3.5.4-8.iso-img`
* It will continue to boot with loading the actual "boot/fdi-image/initrd0.img" system, will fail to mount the squashfs and the drop to dracut shell.
* Screenshots and rdsosreport from test environments will be attached shortly.
* Use a "grubx64.efi" from older "grub2-efi-x64" and the discovery works just fine.

Expected results:
* System should be getting discovered and build properly.


Additional info:
* This only affects the EFI based discovery
* Under the same scenario, if we create a EFI based system with simple network-based deployment that will work just fine.
Comment 1 Sayan Das 2020-08-19 08:52:42 UTC 
Created attachment 1711823 [details]
First error
Comment 2 Sayan Das 2020-08-19 08:53:14 UTC 
Created attachment 1711824 [details]
Second error
Comment 12 Lukas Zapletal 2020-08-24 10:57:45 UTC 
Workaround: Downgrade grub2 and run installer or extract and replace /var/lib/tftpboot/grub2/grubx64.efi file from grub2-efi-x64 RPM package. For quick download here are all builds (RPM, EFI files) so you can test with any of the versions mentioned above (x86_64):

http://people.redhat.com/~lzapleta/grub/
Comment 15 Lukas Zapletal 2020-08-24 14:19:18 UTC 
WORKAROUND WITH HOTFIX:

http://people.redhat.com/~lzapleta/grub/grub2-efi-x64-cdboot-2.02-0.87.el7.hotfix/grubx64.efi

SecureBoot will not work as this file is not signed.
Comment 21 michael.mcconachie 2020-10-22 20:54:37 UTC 
@lzap - backporting /var/lib/tftpboot/grub2/grubx64.efi from the file in the RPM you shared fixed our CentOS7 Foreman 2.14 instance that was also unable to make use of the fdi image on UEFI enabled hosts. THANK YOU!
Comment 22 Lukas Zapletal 2020-10-23 07:11:24 UTC 
Nice, I am assuming you don't have any additional request and the NEEDINFO flag was a mistake. Thanks!

For the record, you can't go wrong with Rawhide version: https://archives.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/x86_64/os/EFI/BOOT/grubx64.efi
Comment 24 Roman Plevka 2021-02-23 14:08:38 UTC 
FAILQA
for 6.9.0-14.0

using grub2-efi-x64-2.02-0.87.el7.x86_64
GRUB seems to be unable to process the default grub.cfg file.

using a default workflow with booting an unknown host, i can see the EFI requesting and retrieving grubx64.efi and grub.cfg but it falls back to the grub shell.
Comment 25 Lukas Zapletal 2021-02-23 14:32:14 UTC 
We are hit by a different but, we commited a change that added

echo blah blah can't blah blah

which obviously breaks grub2 parser and it goes into a black screen. Ouch.

Please cherry pick this trivial fix: https://github.com/theforeman/foreman/pull/8317/files

Thanks
Comment 27 Javier Martinez Canillas 2021-02-23 14:36:34 UTC 
(In reply to Roman Plevka from comment #24)
> FAILQA
> for 6.9.0-14.0
> 
> using grub2-efi-x64-2.02-0.87.el7.x86_64

Did you try with the grub2-efi-x64-cdboot-2.02-0.87.el7.x86_64 package (note the
-cdboot suffix before the package NVR), this sets GRUB prefix to /EFI/BOOT instead
of the /EFI/redhat/ path.

> GRUB seems to be unable to process the default grub.cfg file.
> 
> using a default workflow with booting an unknown host, i can see the EFI
> requesting and retrieving grubx64.efi and grub.cfg but it falls back to the
> grub shell.

What's the path of grub.cfg requested ?
Comment 28 Roman Plevka 2021-02-23 14:45:00 UTC 
(In reply to Lukas Zapletal from comment #25)
> We are hit by a different but, we commited a change that added
> 
> echo blah blah can't blah blah
> 
> which obviously breaks grub2 parser and it goes into a black screen. Ouch.
> 
> Please cherry pick this trivial fix:
> https://github.com/theforeman/foreman/pull/8317/files
> 
> Thanks

I can confirm that https://github.com/theforeman/foreman/pull/8299 fixes the issue.
Comment 30 Roman Plevka 2021-03-10 13:56:58 UTC 
VERIFIED
on sat6.9.0-16.0

grub config is now valid and is loaded properly.
Comment 33 errata-xmlrpc 2021-04-21 13:17:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313
Note You need to log in before you can comment on or make changes to this bug.