Bug 1970632

Summary: boot fails when bootloader is \EFI\BOOT\BOOTX64.EFI
Product: Red Hat Enterprise Linux 8 Reporter: Bob Fournier <bfournie>
Component: shimAssignee: Bootloader engineering team <bootloader-eng-team>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.4CC: amccrae, anujmaurya, arukum, augol, bdonahue, bfournie, bgilbert, bootloader-eng-team, bpeck, brault, cavery, decui, derekh, dgilmore, dkinkead, dtantsur, fkrska, fmartine, frank.ramsay, glenn.teachey, grajaiya, gteachey, jaredz, jkoten, jkreger, KurtDavidson, lmiksik, lshilin, lszubowi, mamccoma, matthew.mcgovern, mgourin, mheslin, miabbott, mknutson, mmorsy, nchoudhu, nkononov, ovasik, peterm, pjanda, pjones, racedoro, release-test-team-automation, rharwood, rmetrich, rondom, saimsh, sbaker, sbarcomb, sdodson, sraskar, tsedovic, vkuznets, walters, xuli, xxiong, yacao, yjoseph, ykashtan, yuxisun
Target Milestone: rcKeywords: OtherQA, Triaged
Target Release: 8.6Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: 15.5-2.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1966129
: 2020259 2020260 2060846 (view as bug list) Environment:
Last Closed: 2022-05-10 15:33:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1966129, 1971018, 1972213, 1973314, 1976074, 1976079    
Bug Blocks: 1978547, 2007040, 2014472, 2020259, 2020260, 2024217, 2038916, 2060846    
Attachments:
Description Flags
Screenshot of the microsoft hardware partner center dashboard. none

Comment 3 Tomas Sedovic 2021-06-11 11:37:53 UTC
This has been created as a clone of the https://bugzilla.redhat.com/show_bug.cgi?id=1966129 issue.

1966129 now tracks a workaround in Ironic that will let us avoid this issue.

This BZ is for the underlying issue in RHEL/shim.

Comment 4 Bob Fournier 2021-07-15 15:33:13 UTC
Any update on this? We modified the bootloader that Ironic sets to use the CSV file (so shim.efi) when installing the image to get past this - see https://bugzilla.redhat.com/show_bug.cgi?id=1966129. However, on the server in this bug - https://bugzilla.redhat.com/show_bug.cgi?id=1978547#c13, the UEFI boot order is changed when set to boot HDD causing BOOTX64.EFI apparently to be used, and the boot to fail.

Comment 5 Steve Baker 2021-07-20 20:03:33 UTC
(In reply to Bob Fournier from comment #4)
> Any update on this? We modified the bootloader that Ironic sets to use the
> CSV file (so shim.efi) when installing the image to get past this - see
> https://bugzilla.redhat.com/show_bug.cgi?id=1966129. However, on the server
> in this bug - https://bugzilla.redhat.com/show_bug.cgi?id=1978547#c13, the
> UEFI boot order is changed when set to boot HDD causing BOOTX64.EFI
> apparently to be used, and the boot to fail.

This particular issue is not related to the shim, but when the shim is fixed this issue will be hidden again. The BMC is changing the boot entry order when the boot mode is changing, which it shouldn't be. We'll discuss that in bug #1978547.

Comment 6 Javier Martinez Canillas 2021-08-20 10:12:39 UTC
*** Bug 1993083 has been marked as a duplicate of this bug. ***

Comment 32 Renaud Métrich 2021-10-05 08:54:04 UTC
Dear all, please kindly review my PR in Upstream shim: https://github.com/rhboot/shim/pull/422

This is a replacement for the commit that introduced duplicated entries.

I tested on Dell R740 with success, but please carefully review the code just in case I'm wrong somewhere.

Comment 36 Derek Higgins 2021-10-15 17:07:06 UTC
With a shim package including the PR from Renaud I replaced shim
related files in a RHCOS image, this reliably fixed the issue we
had been having in OCP ipi. Once a official package is ready I can
also verify it.

Comment 38 Petr Janda 2021-10-15 19:45:53 UTC
Providing QA ACK based on comment 36 and comment 37.

for QE: pre-verification can be done with unsigned build as the issue is not related to SecureBoot

Comment 57 anujmaurya 2022-01-21 08:57:37 UTC
Any update on the testing of RC2 shim package? who is testing this?

Comment 60 Mark Heslin 🎸 2022-02-05 05:05:03 UTC
Hi All,

Can we clarify where this stands currently. There seems to be a bit of confusion. 
Is RH waiting on MS to sign the SHIM or has that already been done? 

-m

Comment 62 Arun 2022-02-11 21:52:50 UTC
(In reply to Mark Heslin from comment #60)
> Hi All,
> 
> Can we clarify where this stands currently. There seems to be a bit of
> confusion. 
> Is RH waiting on MS to sign the SHIM or has that already been done? 
> 
> -m

Hi Mark,

Posting the comment below from Kurt as he is currently not having access to Bugzilla.

"Your post implies that the fix is ready and was sent to MS. I want to confirm our understanding and ensure that we’re aligned with Red Hat. Our understanding is that the Build is still in testing and has not been provided to MS to sign. Can you please advise on whether we should have the fix already? Per the MS dev contact that we are working with, we don’t have this yet and he also understands that the shim is still in RH QA."

Thanks
Arun

Comment 63 Glenn 2022-02-14 19:26:37 UTC
Impacted customer for this on our end(Microsoft) is HDI.
This is causing problems with the ability to deploy RHEL 8 VMs. 

It would be best if we're able to give them a hard date/month shim 15.5 is going to be officially available, as this has been ongoing for 8 months now.

Comment 69 Mark Heslin 🎸 2022-02-23 21:25:41 UTC
+Ron Dominguez, Glenn Teachey are looking for update from RH - moved to here in BZ:

>I see the latest update on the link is a ‘needsinfo’, but we still don’t have a ballpark for when to expect the SHIM to release.
>
>We had previously been told sometime in Jan 2022, which had been passed on to the customer as a possible timeframe for us 
>to get the shim and be able to sign it.
>
>Are you able to advise if there’s any updates we can provide to the customer?

Comment 71 saimsh-msft 2022-03-01 19:55:22 UTC
Hi all, 
This case has been open to MS for over 8 months now and the customer has been growing impatient waiting for the fix to be released. 
This has hurt both MS and Redhat PR with the customer. 
we are currently in March and the customer was provided an ETA of January.
we would like to have a fixed date for the official release of fix to set correct expectations with the customer. 
Thanks,
Saim

Comment 72 Bertrand 2022-03-02 14:31:12 UTC
Does anyone know where I could fetch the Shim 15.5 to be tested by INTEL in the context of RHEL 8.4?

Comment 73 KurtDavidson 2022-03-04 22:58:54 UTC
Jared Dominguez - I understand that the bootloader team is looking for ISV's to test the shim. Is it possible for our joint customer to get a copy of the shim to test? If so, what steps would be needed to arrange this?

Thanks.
Kurt

Comment 74 Jared Dominguez 2022-03-07 15:07:00 UTC
We were able to get enough testing of the RC candidate but are still waiting on testing of the RHEL build candidates before sending them for the shim review and signing processes. A partner engineer will have access to them here:
RHEL 8 - https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=43495224
RHEL 9 - https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=43494819

Comment 75 anujmaurya 2022-03-22 04:52:41 UTC
hi Jared,

Do we need to test SHIM 15.5 or the Shim 15.5 RC2 candidate? what's the process to test the shim?
Is this the correct source https://github.com/rhboot/shim/releases/tag/15.5?

Comment 77 Jared Dominguez 2022-03-22 18:37:07 UTC
We submitted shim 15.5 (post-review) to Microsoft for signing on March 10th, so as soon as we get it back, we can finish packaging and provide an update.

Comment 78 anujmaurya 2022-03-23 14:45:35 UTC
Hi Jared,

Thanks for the confirmation. To whom did you share the Shim for signing? did you use any specific portal for raising the signing request?

Thanks,
Anuj Maurya

Comment 79 Peter Jones 2022-03-23 15:06:36 UTC
Created attachment 1867850 [details]
Screenshot of the microsoft hardware partner center dashboard.

They are submissions 13687950037624770 , 14240784021361452 , and 13516179698635100 on https://partner.microsoft.com/en-us/dashboard/hardware/filesign .

Comment 83 Arun 2022-04-01 18:35:51 UTC
@pjones the submissions 13687950037624770 , 14240784021361452 , and 13516179698635100 are showing in failed state. In order to determine the root cause of this failure, Microsoft HDC team has requested you to raise an SR by following below steps and the issue will be addressed with priority : 

1.	Go here  http://aka.ms/hardwaresupport
2.	Click Contact Us
3.	Make sure you are signed in with a user associated with the HDC account in Partner Center
4.	Select Chat now or submit an incident to receive email support: “

Comment 86 Jared Dominguez 2022-04-07 13:55:30 UTC
There was an issue with Microsoft's signing portal. We resubmitted the signing requests earlier this week. There is no alternative (other than sticking to the current RHEL shim packages) if we can't get the updated shim into the RC.

Comment 87 Bertrand 2022-04-13 09:36:13 UTC
Hi Jared, 
When this is signed by Microsoft, does it clear the way to have it included in RHEL 8.6 and subsequently in RHEL 8.4?
Are we still gated by the lack of testing (OtherQA)?

Thanks,
Bertrand

Comment 97 anujmaurya 2022-04-27 17:55:01 UTC
Folks, Just checked with the HDC team from MS, the shim is already signed on 14/04. 
Seems like it will be released with RHEL 8.6. What are the timelines for releasing the SHIM updates for older 8.x images?

When can we expect it to be delivered via RHUI?

Thanks,
Anuj Maurya

Comment 100 Arun 2022-05-04 16:55:57 UTC
Hi RedHat Team,

Is there any ETA for these updates to be available in RHUI and for older 8.X images?

Thank you
Arun

Comment 101 Petr Janda 2022-05-09 09:30:12 UTC
*** Bug 2072575 has been marked as a duplicate of this bug. ***

Comment 104 errata-xmlrpc 2022-05-10 15:33:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (shim bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2126

Comment 105 Derek Higgins 2022-05-18 08:35:23 UTC
Already verified