Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1768689

Summary: grub2-install fails with "efibootmgr failed to register the boot entry: Unknown error 21991.'
Product: Red Hat Enterprise Linux 8 Reporter: Bob Fournier <bfournie>
Component: grub2Assignee: Bootloader engineering team <bootloader-eng-team>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.1CC: fmartine, pjanda, zveleba
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: grub2-2.02-80.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-28 16:57:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1459187, 1767085    

Description Bob Fournier 2019-11-05 02:05:37 UTC
Description of problem:

This is with Openstack (OSP-16) ironic-python-agent, it calls grub2-install when using UEFI here [1].  This error occurs every time although the error code changes (I have seen 22001, 21991, and 22051 on different runs).  The grub2 being used is 2.02-74.  

I believe it is hitting this issue - https://github.com/rhboot/grub2/commit/b71ac53751877dac18cfc091db2e22b805462278which is fixed but I'm not sure if the fix is in 2.02.  The error message and the fact that the error code changes is consistent with the bug description. The error code is changing because the return is an uninitialized variable.

[1] https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/extensions/image.py#L195

Version-Release number of selected component (if applicable):

grubby-8.40-37.el8.x86_64
grub2-pc-modules-2.02-74.el8_0.1.noarch
grub2-tools-2.02-74.el8_0.1.x86_64
grub2-common-2.02-74.el8_0.1.noarch
grub2-pc-2.02-74.el8_0.1.x86_64
grub2-tools-minimal-2.02-74.el8_0.1.x86_64
grub2-tools-extra-2.02-74.el8_0.1.x86_64


How reproducible:

The ironic-python-agent (IPA) writes the overcloud image to disk and then runs grub2-install.  This fails every time and prevents the image from booting.


Steps to Reproduce:
1.  Deploy an image using openstack which will load IPA, copy the overcloud image, and run grub2-install

Actual results: 
grub2-install fails and overcloud image can't boot


Expected results:
No grub2-install errors and image boots.


Additional log messages:
Nov 04 15:15:34 hardprov-per430-01.snedlab.lab.eng.rdu2.redhat.com ironic-python-agent[1608]: 2019-11-04 15:15:33.230 1608 ERROR root [-] Command execution error: Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command.
                                                                                              Command: chroot /tmp/tmp_gv1majs /bin/sh -c "grub2-install /dev/sda"
                                                                                              Exit code: 1
                                                                                              Stdout: ''
                                                                                              Stderr: 'Installing for x86_64-efi platform.\ngrub2-install: error: efibootmgr failed to register the boot entry: Unknown error 21991.\n'.: ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command.
                                                                                              Command: chroot /tmp/tmp_gv1majs /bin/sh -c "grub2-install /dev/sda"
                                                                                              Exit code: 1
                                                                                              Stdout: ''
                                                                                              Stderr: 'Installing for x86_64-efi platform.\ngrub2-install: error: efibootmgr failed to register the boot entry: Unknown error 21991.\n'.

Comment 1 Bob Fournier 2019-11-05 02:08:42 UTC
Note that the link to the upstream commit above got munged, it is  https://github.com/rhboot/grub2/commit/b71ac53751877dac18cfc091db2e22b805462278.

Comment 2 Bob Fournier 2019-11-06 18:28:06 UTC
It looks like the bug still exists in 2.02-78. I checked grub2-2.02-78.el8.src.rpm and see in release-to-master.patch that this fix introduced the uninitialized rc value.

diff --git a/grub-core/osdep/unix/platform.c b/grub-core/osdep/unix/platform.c
index a3fcfcacaa814d3ab62104f0dd406ef0c2163613..ca448bc11a05b9e0c6203a799ff62ab1dd75274f 100644
--- a/grub-core/osdep/unix/platform.c
+++ b/grub-core/osdep/unix/platform.c
@@ -78,19 +78,20 @@ get_ofpathname (const char *dev)
                   dev);
 }

-static void
+static int
 grub_install_remove_efi_entries_by_distributor (const char *efi_distributor)
 {
   int fd;
   pid_t pid = grub_util_exec_pipe ((const char * []){ "efibootmgr", NULL }, &fd);
   char *line = NULL;
   size_t len = 0;
+  int rc;  <=  This is the problem


Any idea when fix https://github.com/rhboot/grub2/commit/b71ac53751877dac18cfc091db2e22b805462278 will be backported?

Comment 3 Bob Fournier 2019-11-13 13:38:38 UTC
*** Bug 1767085 has been marked as a duplicate of this bug. ***

Comment 4 Alberto Ruiz 2019-11-20 14:57:07 UTC
Peter Jones might have a better idea as to what is going wrong, but in the meantime.

Can you verify if /sys is bind mounted inside the chroot and that is rw?

Comment 5 Javier Martinez Canillas 2019-11-20 15:41:36 UTC
There is certainly a bug in grub2-install for EFI, that fails to create Boot Entries when there are none.

The fix is correct and we can cherry-pick it but I'm not sure if that will solve your issue. For me this is just a symptom of Openstack's ironic-python-agent calling the grub2-install tool on EFI. This is not a good idea for the following reasons:

1- It calls grub2-mkimage which re-links the grub2 EFI binary and so this is not signed. That means that the machine won't boot with Secure Boot enabled.
2- It removes the BootEntry that was created during RHEL installation, and creates a new one that sets the file path to the unsigned grub2 EFI binary instead of shim. So again the machine will fail to boot with Secure Boot.
3- The grub2-install tool isn't that tested on EFI, so there are bugs like the one mentioned in this bugzilla. We can fix them but you will still face issues (1) and (2).

For this particular bug, I will do a scratch build that cherry-picks the mentioned upstream commit for you to test. But I think that the correct fix is to not call grub2-install for EFI systems.

Comment 7 Bob Fournier 2019-11-20 15:57:22 UTC
Thanks a lot Javier.  I will try the scratch build. 

Yes, when secure boot is enabled we need to fix IPA to not use grub2-install, we have an open bug for that - https://storyboard.openstack.org/#!/story/2006847.  We shouldn't have an issue when using grub2-install in the non-secure boot case but we're hitting this.

For the question above, below is the mount output on the IPA after grub2-install is called and this is how its invoked in IPA - https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/extensions/image.py#L251

[root@hardprov-per430-01 ~]# mount
rootfs on / type rootfs (rw,size=32302388k,nr_inodes=8075597)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=32663276k,nr_inodes=8165819,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=49698)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6586420k,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

Comment 8 Javier Martinez Canillas 2019-11-20 16:10:52 UTC
Hello Bob,

(In reply to Bob Fournier from comment #7)
> Thanks a lot Javier.  I will try the scratch build. 
> 
> Yes, when secure boot is enabled we need to fix IPA to not use
> grub2-install, we have an open bug for that -
> https://storyboard.openstack.org/#!/story/2006847.  We shouldn't have an
> issue when using grub2-install in the non-secure boot case but we're hitting
> this.
>

Yes, I'm all for fixing this particular issue but I'm not sure that I agree with you that calling grub2-install in the non-SB case is OK.

Since that means that you will prevent SB to be enabled. In other words, with the EFI Boot entries that are created during installation or by the EFI fallback mechanism, you could either boot with Secure Boot enabled or disabled.

But if you call grub2-install, you won't have shim anymore in the boot path nor have a signed grub EFI binary (even if grub2-install doesn't have any bugs). So at this point SB can't be enabled anymore.

Comment 9 Bob Fournier 2019-11-24 20:17:43 UTC
I've verified that the backport for [1] to add the variable init does fix the grub2-install issue I was hitting.  I think the scratch build at
http://brew-task-repos.usersys.redhat.com/repos/scratch/fmartine/grub2/2.02/79.el8_1/ was replaced by a build with latest CVE fix on 11/22, but I was
able to test the patch in my own scratch build. 

And yes, I agree that for all case of enabling secure boot we'll need to change the IPA code to call efibootmgr directly.

Thanks.

[1] https://github.com/rhboot/grub2/commit/b71ac53751877dac18cfc091db2e22b805462278

Comment 10 Petr Janda 2019-11-27 13:55:41 UTC
Hello Bob, are you willing to test it again once there is final build in compose, please. If so, I will provide qa_ack relying on you.

Comment 11 Bob Fournier 2019-11-27 16:18:48 UTC
Petr - yes, I have a setup that can test this when the final build is ready, just give me the link when available.

Comment 12 Petr Janda 2019-11-28 10:48:27 UTC
Thank you Bob, based on comment 11 providing qa_ack

Comment 14 Bob Fournier 2019-11-29 23:32:02 UTC
With grub2-2.02-80.el8, I verified I'm no longer seeing the "efibootmgr failed to register the boot entry: Unknown error" in the ironic-python-agent logs and the server boots correctly.

Comment 15 Zdenek Veleba 2020-03-16 12:19:35 UTC
Changing status to Verified based on information from Bob Fournier.

Comment 17 errata-xmlrpc 2020-04-28 16:57:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1869