Bug 1993083

Summary:	Shim breaks first reboot from console on new builds in Azure 8.4
Product:	Red Hat Enterprise Linux 8	Reporter:	Parikshit Khedekar <pkhedeka>
Component:	shim	Assignee:	Javier Martinez Canillas <fmartine>
Status:	CLOSED DUPLICATE	QA Contact:	Release Test Team <release-test-team-automation>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	8.4	CC:	cavery, falim, fmartine, huzhao, jiazhang, matthew.mcgovern, mmorsy, pjones, ribarry, rmetrich, sbarcomb, snejoshi, vkuznets, xuli, xxiong, yacao, yuxisun
Target Milestone:	beta
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-08-20 10:12:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Parikshit Khedekar 2021-08-12 10:52:12 UTC

Description of problem:

The VM provisioned for RHEL 8.4 in Azure will fail to do a first reboot if
it's done from Azure portal. Post troubleshooting and checking all the things
as kdump,WaLinuxAgent customer found the shim package is responsible for
this.

How reproducible:

Always post provisioning the VM and rebooting it from portal with a restart
switch/button.

Steps to Reproduce:

[1] Get the Azure Cli and provision a VM as,

$az login
$az account set --subscription <your sub id>
$az group create -l <location (I prefer westus2)> -n <Resource Group name>
$az vm create --name <VM Name> --resource-group <RG Name that is create by above command> --location <westus2> --image "RedHat:RHEL:84-gen2:8.4.2021053102" --size Standard_D2s_v3 --admin-username <username> --admin-password <password>

Secondly, you can go through this link for reference :
https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/redhat/redhat-images

[2] Once done Open the Azure portal and click the restart button. The VM will
notify about rebooting but will never come up.

[3] Upon hard - reset or restart as Stop and Start the VM will start and there
after the "restart" will work as expected.

*** The issue occurs only at first reboot done using restart button" ***

Version-Release number of selected component (if applicable):
shim-x64-15.4-2.el8_1.x86_64

Actual results:

System fails to reboot with nothing printed to console as
suspect, no screenshot as there is no direct console and so far I see it
fails to return the boot status to arm signal sent by the webcode to
hypervisor.

Hard reset boots it up and next "restarts" work fine so shims seems to be
adapting something.

Expected results:
Shall not fail to reboot at first reboot from Portal call.

Additional info:

Customer did check as,

We have

RHEL 8.4 shim-x64-15.4-2.el8_1.x86_64
RHEL 8.3 shim-x64-15-16.el8.x86_64

So even if we update the 8.3 release with shim-x64-15.4-2.el8_1.x86_64 it will
fail to boot once. So I suspect the arm calls from portal are failing as
while checking this more I have checked journal and can see a kill sent as
normal shutdown.

Limited to Hyper-V env so can't have direct serial console logs and I have
asked to share them, will forword if get available.

Downgrading the shim to shim-x64-15-16.el8.x86_64 on RHEL 8.4 image makes the
issue go away!

Comment 7 Peter Jones 2021-08-17 17:47:56 UTC

At present I cannot get to a point where I can even try to reproduce this issue.  While we work that out, anyone who *can* reproduce the issue, please:

1) install shim-x64-15-16.el8.x86_64
2) from linux, run "mokutil --set-verbosity true" as root
3) if possible, copy \EFI\redhat to \EFI\test , put the shimx64.efi from shim-x64-15.4-2.el8_1.x86_64 there, and set it up as a second boot entry in the system firmware
4) capture the console log while restarting with this new boot entry selected

Comment 11 Yuxin Sun 2021-08-18 10:49:18 UTC

(In reply to Peter Jones from comment #7)
> At present I cannot get to a point where I can even try to reproduce this
> issue.  While we work that out, anyone who *can* reproduce the issue, please:
> 
> 1) install shim-x64-15-16.el8.x86_64
> 2) from linux, run "mokutil --set-verbosity true" as root
> 3) if possible, copy \EFI\redhat to \EFI\test , put the shimx64.efi from
> shim-x64-15.4-2.el8_1.x86_64 there, and set it up as a second boot entry in
> the system firmware
> 4) capture the console log while restarting with this new boot entry selected

Hi Peter,

If I run "mokutil --set-verbosity true" and then restart the VM, the VM shows a "Secure Boot not enabled" blue screen, and because Azure only have serial console, it cannot get passed.
Besides, Azure cannot select boot order, so I have to set the new entry as the first boot order.

Comment 12 Renaud Métrich 2021-08-18 11:36:55 UTC

Hi pjones,
Setting verbosity indeed drops you to a OK prompt, so that's problematic since console is not available.
Would it be possible to provide a shim-x64 package that doesn't show that OK prompt?