Bug 2211029

Summary:

Shim 15.6-3 update breaks NEC/Fujitsu UEFI boots on certain hardware

Product:

Red Hat Enterprise Linux 7

Reporter:

Klaas Demter <klaas>

Component:

shim

Assignee:

Bootloader engineering team <bootloader-eng-team>

Status:

CLOSED MIGRATED

QA Contact:

Release Test Team <release-test-team-automation>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

7.9

CC:

amepatil, chorn, fqi, hmatsumo, hshuai, jaredz, jlinton, kueda, mlewando, pjanda, pjones, qguo, raravind, rhayakaw, rmetrich, samukawa-oxa, sbarcomb, tatsu-ab1, tumeya, yidliu, yoguma, yuhigash, yusokada

Target Milestone:

Keywords:

MigratedToJIRA

Target Release:

---

Flags:

klaas: needinfo-
pm-rhel: mirror+

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2023-09-16 20:07:13 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1817782

Attachments:

Description	Flags
picture of debug output	none
picture of debug output hang	none
Archive of screenshots extracted from the videos in case	none
New screenshots taken with the rpm from case	none
boot error message on RHEL8	none

Description Klaas Demter 2023-05-30 10:26:09 UTC

Description of problem:
it seems the current rhel 7.9 shim release breaks our fujitsu rx2540 m2 servers. I would guess from the bz discussion around https://bugzilla.redhat.com/show_bug.cgi?id=2113005 it could be this issue :)


Version-Release number of selected component (if applicable):
shim-x64-15.6-3.el7_9.x86_64


How reproducible:
Have a fujitsu rx2540 m2 bios version V5.0.0.11 R1.31.0 


Steps to Reproduce:
1. Install RHEL 7.9
2. Update to shim-x64-15.6-3.el7_9.x86_64
3. Reboot

Actual results:
Hang

Expected results:
Boots


Additional info:
Support Case 03525053

Comment 3 Klaas Demter 2023-05-30 13:09:15 UTC

Last line of debug log when booting with current shim version:
mok.c:298:mirror_one_esl() 00001D58 21 0f 67 cl XX XX XX XX XX XX XX XX XX XX XX XX |!.g.|

Comment 4 Marta Lewandowska 2023-05-30 13:59:52 UTC

Based on that last line it could be this one: https://bugzilla.redhat.com/show_bug.cgi?id=2209291

It looks like you know how to turn on mokutil verbosity. Could you please attach the full log?

Comment 5 Klaas Demter 2023-05-30 14:32:26 UTC

(In reply to Marta Lewandowska from comment #4)
> Based on that last line it could be this one:
> Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2209291
> 
> It looks like you know how to turn on mokutil verbosity. Could you please
> attach the full log?

Hi,
I can't view that bug, it's private (or partner only). I do not have a text log, but I've attached the full video of debug output to the support case (multple hundred MB).

Greetings
Klaas

Comment 7 Marta Lewandowska 2023-05-30 19:06:03 UTC

Hi Klaas,
you should have access to bz#2209291 now.

Comment 8 Marta Lewandowska 2023-05-30 19:38:13 UTC

(In reply to Klaas Demter from comment #5)
> (In reply to Marta Lewandowska from comment #4)
> > Based on that last line it could be this one:
> > Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2209291
> > 
> > It looks like you know how to turn on mokutil verbosity. Could you please
> > attach the full log?
> 
> Hi,
> I can't view that bug, it's private (or partner only). I do not have a text
> log, but I've attached the full video of debug output to the support case
> (multple hundred MB).
> 
> Greetings
> Klaas

you should have access now. I looked at some of your videos and it very much looks like it's the same bug.

Comment 9 Klaas Demter 2023-05-31 06:24:31 UTC

Hi,
yes looking at the output in the private bug I agree. I'm attaching two screenshots of my debug video which should contain the most important information.

I would suggest to continue working in the public bug (this one), so other customers can also find it if they search like me :)


The other bug thinks this is CPU related, so I'll also post the information I have gathered, but I only have a very limited test-set, I only have a couple of servers this old left :)


Not working:
Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (Broadwell EP)
Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz (Broadwell EP)

Working:
Intel(R) Xeon(R) CPU E7-8880 v3 @ 2.30GHz (Haswell EX)


Greetings
Klaas

Comment 10 Klaas Demter 2023-05-31 06:27:18 UTC

Created attachment 1968003 [details]
picture of debug output

Comment 11 Klaas Demter 2023-05-31 06:27:54 UTC

Created attachment 1968004 [details]
picture of debug output hang

Comment 12 Klaas Demter 2023-05-31 08:04:02 UTC

I also have another old broadwell test system, but it's running RHEL8

Dell PowerEdge R730 Intel(R) Xeon(R) CPU E5-2620 v4 (Broadwell EP) Bios 2.16.0
shim-x64-15.6-1.el8.x86_64

works fine, both with secureboot enabled and disabled.

Comment 13 Renaud Métrich 2023-05-31 09:49:30 UTC

Klaas confirmed (in the associated case) that RHEL8 and RHEL9 Shim (15.6-1) work fine, which tends to indicate an issue with RHEL7 branch, which seems more recent than RHEL8 and RHEL9 ones.

Comment 14 Renaud Métrich 2023-05-31 10:11:04 UTC

*** Bug 2209291 has been marked as a duplicate of this bug. ***

Comment 15 Marta Lewandowska 2023-05-31 11:11:38 UTC

If this is contingent on CPU, then in https://bugzilla.redhat.com/show_bug.cgi?id=2209291#c5 they mentioned that the bug did not manifest on B120g-h      E5-2620 v4(Broadwell-EP) so it's hard to tell whether it's the shim version or something else, unfortunately.

Comment 16 raravind 2023-05-31 11:32:00 UTC

Marta / Renaud ,

Is it worth to approach microcode team as the issue didnt occur in some of the CPU's based on https://bugzilla.redhat.com/show_bug.cgi?id=2209291#c5?

Comment 17 Klaas Demter 2023-05-31 13:20:05 UTC

So the servers that do not work here are:


Manufacturer: FUJITSU
Product Name: PRIMERGY RX2540 M2
Version: GS01

I have two different CPU types (older machines have *97, later ones have *98)
Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (Broadwell EP)
Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz (Broadwell EP)

They all show the same microcode:
microcode: sig=0x406f1, pf=0x1, revision=0xb000040

Bios:
Vendor: FUJITSU // American Megatrends Inc.
Version: V5.0.0.11 R1.31.0 for D3289-B1x
(which is the latest one released by Fujitsu https://support.ts.fujitsu.com/DownloadManager/globalflash/SystemBoard/D3289-RX2540M2/ )

Comment 18 Klaas Demter 2023-05-31 13:52:54 UTC

And I tried shim-x64-15.6-3.el7_9 on the Dell Broadwell server I have, it works there.

Manufacturer: Dell Inc.
Product Name: PowerEdge R730


Intel(R) Xeon(R) CPU E5-2620 v4 (Broadwell EP)
microcode: sig=0x406f1, pf=0x1, revision=0xb000040

Bios:
Vendor: Dell Inc.
Version: 2.16.0
Release Date: 07/20/2022

works there with secure boot enabled and disabled.

Comment 19 Marta Lewandowska 2023-05-31 15:05:30 UTC

Looking at the failure logs provided, and comparing to a successful boot, it looks like when the boot is unsuccessful, it is because after reading in the first variable (MokList) and then copying it, the resulting data size is too large / allotted memory too small.

mok.c:763:mirror_one_mok_variable() calling mirror_mok_db("MokListRT",  datasz=34240)
mok.c:235:get_max_var_sz() calling RT->QueryVariableInfo() at 0x7FAE67DC
mok.c:251:get_max_var_sz() max_var_sz:6F40 remaining_sz:6F5A max_storage_sz:10000

datasz is 34240 (= 0x85C0) while max_var_sz is 0x6F40 (0x651B in movie) and so rather than returning Success after reading in like this: 

mok.c:763:mirror_one_mok_variable() calling mirror_mok_db("MokListRT",  datasz=34240)
mok.c:235:get_max_var_sz() calling RT->QueryVariableInfo() at 0x8D64B20C
mok.c:251:get_max_var_sz() max_var_sz:87C0 remaining_sz:87FC max_storage_sz:FFE4
mok.c:768:mirror_one_mok_variable() mirror_mok_db("MokListRT",  datasz=34240) returned Success
mok.c:812:mirror_one_mok_variable() returning Success
mok.c:925:import_one_mok_state() returning Success

and then moving on to import the next variable, it is chopping the MokList up into smaller variables so that it can read in as much as it can, hence the numbering.
It appears to read in two MokListRTs successfully, and should have enough space for the third one, but then it hangs, and I'm not sure why yet.

Comment 20 Marta Lewandowska 2023-06-01 14:39:17 UTC

Since the cause of this might be running out of memory or space to store efi variables, I would ask for the output of the following commands:
# lsblk -f /boot/efi
# ls -l /sys/firmware/efi/efivars/
# free

and possibly running memmap from the EFI Shell.

thanks.

Comment 21 Renaud Métrich 2023-06-02 05:35:40 UTC

Hi Klaas,

Marta believes there could be a space issue, could you run on one of your failing system the commands in comment just above and report?
Ideally, execute the commands after booting the failing Shim but also after booting the RHEL8/RHEL9 Shim I gave you in the case.

Honestly, I doubt it's the root cause, since I would expect the UEFI space to be exactly the same whichever Shim release is booted.

Renaud.

Comment 22 Klaas Demter 2023-06-02 08:00:38 UTC

so the first command does not work on the mountpoint, only on the device :)


System A (15.6-3 failing, currently booted with RHEL 9 shim)
# lsblk -f /dev/sda1
NAME FSTYPE LABEL    UUID                                 MOUNTPOINT
sda1 vfat   HANA_EFI 6DE0-29C2                            /boot/efi


# ls -l /sys/firmware/efi/efivars/
total 0
-rw-r--r-- 1 root root  148 May 31 11:21 Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  154 May 31 11:21 Boot0001-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  154 May 31 11:21 Boot0002-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  172 May 31 12:32 Boot0003-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  164 May 31 12:34 Boot0004-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot000F-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0010-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0011-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0012-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0013-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0014-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0015-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  230 May 31 11:21 Boot0016-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    6 May 31 11:21 BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    8 May 31 11:21 BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   26 May 31 11:21 BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   38 May 31 11:21 ConIn-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   38 May 31 11:21 ConInDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   40 May 31 11:21 ConOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   40 May 31 11:21 ConOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 9817 May 31 11:21 db-d719b2cb-3d3a-4596-a3bc-dad00e67656f
-rw-r--r-- 1 root root 9824 May 31 11:21 dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  656 May 31 11:21 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f
-rw-r--r-- 1 root root  656 May 31 11:21 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   40 May 31 11:21 ErrOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   40 May 31 11:21 ErrOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   44 May 31 11:21 fjAvailableHddDriveNumbers-d36eafc3-a917-4cae-9417-448072f2f93c
-rw-r--r-- 1 root root    8 May 31 11:21 FPDT_Volatile-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root   12 May 31 11:21 HiiDB-1b838190-4625-4ead-abc9-cd5e6af18fe0
-rw-r--r-- 1 root root 7133 May 31 11:21 KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 7153 May 31 11:21 KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    8 May 31 11:21 LSI_MR_DEVICE_EXPOSURE-5b6a5a3a-7db5-44f4-92a9-f0c162dd6374
-rw-r--r-- 1 root root    6 May 31 11:21 MaximumTableSize-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root  968 May 31 11:21 MokListRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root    5 May 31 11:21 MokListTrustedRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root   80 May 31 11:21 MokListXRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root    8 May 31 11:21 MonotonicCounter-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root   12 May 31 11:21 OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 1125 May 31 11:21 PK-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 1132 May 31 11:21 PKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   10 May 31 11:21 PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   10 May 31 11:21 PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    8 May 31 11:21 RTC-378d7b65-8da9-4773-b6e4-a47826a833e1
-rw-r--r-- 1 root root    8 May 31 11:21 RTC-378d7b65-8da9-4773-b6e4-a47826a833e2
-rw-r--r-- 1 root root   12 May 31 11:21 S3SS-4bafc2b4-02dc-4104-b236-d6f1b98d9e84
-rw-r--r-- 1 root root   22 May 31 11:21 SbatLevelRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root    5 May 31 11:21 SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    8 May 31 11:21 SetUpdateCountVar-81c76078-bfde-4368-9790-570914c01a65
-rw-r--r-- 1 root root    5 May 31 11:21 SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  132 May 31 11:21 SignatureSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   12 May 31 11:21 SmbiosEntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root   12 May 31 11:21 SmbiosScratchBuffer-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root   12 May 31 11:21 SmbiosV3EntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root    6 May 31 11:21 Timeout-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    5 May 31 11:21 VendorKeys-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   68 May 31 11:21 WriteOnceStatus-4b3082a3-80c6-4d7e-9cd0-583917265df1


# free
              total        used        free      shared  buff/cache   available
Mem:      527416648     6941484   519430948       67812     1044216   519051104
Swap:       2097148           0     2097148


# efibootmgr -v
BootCurrent: 0002
Timeout: 1 seconds
BootOrder: 0000,000F,0010,0011,0012,0013,0014,0015,0016,0001,0002
Boot0000* RedHat Boot Manager   HD(1,GPT,24fedd81-2376-4189-ace8-8f07fa0624a3,0x800,0x200000)/File(\EFI\REDHAT\SHIMX64.EFI)
Boot0001* SHIM RHEL8    HD(1,GPT,24fedd81-2376-4189-ace8-8f07fa0624a3,0x800,0x200000)/File(\EFI\redhat\shim-x64-15.6-1.el8.efi)
Boot0002* SHIM RHEL9    HD(1,GPT,24fedd81-2376-4189-ace8-8f07fa0624a3,0x800,0x200000)/File(\EFI\redhat\shim-x64-15.6-1.el9.efi)
Boot0003* SHIM RHEL7 15.6-3     HD(1,GPT,24fedd81-2376-4189-ace8-8f07fa0624a3,0x800,0x200000)/File(\EFI\redhat\shim-x64-15.6-3.el7_9.efi)
Boot0004* SHIM RHEL7 15-11      HD(1,GPT,24fedd81-2376-4189-ace8-8f07fa0624a3,0x800,0x200000)/File(\EFI\redhat\shim-x64-15-11.el7.efi)
Boot000F* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(b49691019848,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0010* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x0)/Pci(0x0,0x1)/MAC(b4969101984a,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0011* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x0)/MAC(b49691019b64,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0012* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x1)/MAC(b49691019b66,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0013* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(b49691019bac,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0014* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(b49691019bae,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0015* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x0)/MAC(b496910199d8,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0016* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x1)/MAC(b496910199da,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO

ls -al /boot/efi/EFI/redhat/shim*
-rwx------ 1 root root 1252576 Sep 17  2020 /boot/efi/EFI/redhat/shim.efi
-rwx------ 1 root root 1252576 May 31 12:33 /boot/efi/EFI/redhat/shim-x64-15-11.el7.efi
-rwx------ 1 root root  943520 Jun  7  2022 /boot/efi/EFI/redhat/shim-x64-15.6-1.el8.efi
-rwx------ 1 root root  946736 Jun  7  2022 /boot/efi/EFI/redhat/shim-x64-15.6-1.el9.efi
-rwx------ 1 root root  951296 May 31 12:31 /boot/efi/EFI/redhat/shim-x64-15.6-3.el7_9.efi
-rwx------ 1 root root 1252576 Sep 17  2020 /boot/efi/EFI/redhat/shimx64.efi
-rwx------ 1 root root 1246656 Sep 17  2020 /boot/efi/EFI/redhat/shimx64-redhat.efi



System B (failing with 15.6-3, currently booted with shim 15-11.el7)

# lsblk -f /dev/sda1
NAME FSTYPE LABEL    UUID                                 MOUNTPOINT
sda1 vfat   HANA_EFI 6E11-6E32                            /boot/efi

# ls -l /sys/firmware/efi/efivars/
total 0
-rw-r--r-- 1 root root   148 May 29 22:30 Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot0002-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot0004-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot0006-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot0008-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot000A-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot000C-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot000E-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 29 22:30 Boot0010-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     6 May 29 22:30 BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 29 22:30 BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    22 May 29 22:30 BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    38 May 29 22:30 ConIn-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    38 May 29 22:30 ConInDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 29 22:30 ConOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 29 22:30 ConOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  9817 May 29 22:30 db-d719b2cb-3d3a-4596-a3bc-dad00e67656f
-rw-r--r-- 1 root root  9824 May 29 22:30 dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   656 May 29 22:30 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f
-rw-r--r-- 1 root root   656 May 29 22:30 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 29 22:30 ErrOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 29 22:30 ErrOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    44 May 29 22:30 fjAvailableHddDriveNumbers-d36eafc3-a917-4cae-9417-448072f2f93c
-rw-r--r-- 1 root root     8 May 29 22:30 FPDT_Volatile-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root    12 May 29 22:30 HiiDB-1b838190-4625-4ead-abc9-cd5e6af18fe0
-rw-r--r-- 1 root root  7133 May 29 22:30 KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  7153 May 29 22:30 KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 29 22:30 LSI_MR_DEVICE_EXPOSURE-5b6a5a3a-7db5-44f4-92a9-f0c162dd6374
-rw-r--r-- 1 root root     6 May 29 22:30 MaximumTableSize-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 16160 May 29 22:30 MokListRT1-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root     8 May 29 22:30 MonotonicCounter-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root    12 May 29 22:30 OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  1125 May 29 22:30 PK-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  1132 May 29 22:30 PKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    10 May 29 22:30 PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    10 May 29 22:30 PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 29 22:30 RTC-378d7b65-8da9-4773-b6e4-a47826a833e1
-rw-r--r-- 1 root root     8 May 29 22:30 RTC-378d7b65-8da9-4773-b6e4-a47826a833e2
-rw-r--r-- 1 root root    12 May 29 22:30 S3SS-4bafc2b4-02dc-4104-b236-d6f1b98d9e84
-rw-r--r-- 1 root root     5 May 29 22:30 SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 29 22:30 SetUpdateCountVar-81c76078-bfde-4368-9790-570914c01a65
-rw-r--r-- 1 root root     5 May 29 22:30 SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   132 May 29 22:30 SignatureSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    12 May 29 22:30 SmbiosEntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root    12 May 29 22:30 SmbiosScratchBuffer-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root    12 May 29 22:30 SmbiosV3EntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root     6 May 29 22:30 Timeout-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     5 May 29 22:30 VendorKeys-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    68 May 29 22:30 WriteOnceStatus-4b3082a3-80c6-4d7e-9cd0-583917265df1

# free
              total        used        free      shared  buff/cache   available
Mem:      527416648   205642984   295686776    18696292    26086888   301670180
Swap:       2097148           0     2097148


# efibootmgr  -v
BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0002,0006,0008,000A,000C,000E,0010,0004
Boot0000* RedHat Boot Manager   HD(1,GPT,84cd81c8-6d94-4b64-b2c9-7f02b8474c82,0x800,0x200000)/File(\EFI\REDHAT\SHIMX64.EFI)
Boot0002* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(b496910197c4,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0004* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x0)/Pci(0x0,0x1)/MAC(b496910197c6,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0006* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x0)/MAC(b4969101990c,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0008* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x1)/MAC(b4969101990e,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000A* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(b49691019a1c,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000C* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(b49691019a1e,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000E* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x0)/MAC(b49691019bb4,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0010* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x1)/MAC(b49691019bb6,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO


# ls -al /boot/efi/EFI/redhat/shim*
-rwx------ 1 root root 1252576 Sep 17  2020 /boot/efi/EFI/redhat/shim.efi
-rwx------ 1 root root 1252576 Sep 17  2020 /boot/efi/EFI/redhat/shimx64.efi
-rwx------ 1 root root 1246656 Sep 17  2020 /boot/efi/EFI/redhat/shimx64-redhat.efi



System C shim 15.6-3 working:

# lsblk -f /dev/sda1
NAME FSTYPE LABEL    UUID                                 MOUNTPOINT
sda1 vfat   HANA_EFI 34C1-D32F                            /boot/efi

# ls -l /sys/firmware/efi/efivars/
total 0
-rw-r--r-- 1 root root   186 May 28 18:41 Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   202 May 28 18:41 Boot0001-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   202 May 28 18:41 Boot0002-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 28 18:41 Boot0003-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 28 18:41 Boot0004-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 28 18:41 Boot0005-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 28 18:41 Boot0006-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 28 18:41 Boot0007-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   230 May 28 18:41 Boot0008-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   152 May 28 18:41 Boot0009-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   124 May 28 18:41 Boot000A-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     6 May 28 18:41 BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 28 18:41 BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    26 May 28 18:41 BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    38 May 28 18:41 ConIn-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    38 May 28 18:41 ConInDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 28 18:41 ConOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 28 18:41 ConOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root  7765 May 28 18:41 db-d719b2cb-3d3a-4596-a3bc-dad00e67656f
-rw-r--r-- 1 root root    80 May 28 18:41 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f
-rw-r--r-- 1 root root    40 May 28 18:41 ErrOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    40 May 28 18:41 ErrOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 28 18:41 FPDT_Variable-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root    12 May 28 18:41 HiiDB-1b838190-4625-4ead-abc9-cd5e6af18fe0
-rw-r--r-- 1 root root  6205 May 28 18:41 KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 28 18:41 LSI_MR_DEVICE_EXPOSURE-5b6a5a3a-7db5-44f4-92a9-f0c162dd6374
-rw-r--r-- 1 root root     5 May 28 18:41 MemoryOverwriteRequestControl-e20939be-32d4-41be-a150-897f85d49829
-rw-r--r-- 1 root root 17124 May 28 18:41 MokListRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root     5 May 28 18:41 MokListTrustedRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root    80 May 28 18:41 MokListXRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root     8 May 28 18:41 MonotonicCounter-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root    12 May 28 18:41 OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     5 May 28 18:41 PciErrMaskKaw-1a6c6d06-11bf-406b-92c0-256191b8024f
-rw-r--r-- 1 root root  1125 May 28 18:41 PK-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    10 May 28 18:41 PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    10 May 28 18:41 PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     8 May 28 18:41 RTC-378d7b65-8da9-4773-b6e4-a47826a833e1
-rw-r--r-- 1 root root    12 May 28 18:41 S3SS-4bafc2b4-02dc-4104-b236-d6f1b98d9e84
-rw-r--r-- 1 root root    22 May 28 18:41 SbatLevelRT-605dab50-e046-4300-abb6-3dd810dd8b23
-rw-r--r-- 1 root root     5 May 28 18:41 SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     5 May 28 18:41 SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root   132 May 28 18:41 SignatureSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root     6 May 28 18:41 Timeout-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root    12 May 28 18:41 USB_POINT-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root    68 May 28 18:41 WriteOnceStatus-4b3082a3-80c6-4d7e-9cd0-583917265df1


# free
              total        used        free      shared  buff/cache   available
Mem:      527394804   237930252   284080432      470056     5384120   287601968
Swap:       2097148           0     2097148

# efibootmgr -v
BootCurrent: 0009
Timeout: 1 seconds
BootOrder: 0009,0001,0002,0003,0004,0005,0006,0007,0008,000A,0000
Boot0000* Red Hat Enterprise Linux 6    VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0001* UEFI: IP4 Intel(R) Ethernet Controller X540-AT2       /Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(2c600c945cec,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0002* UEFI: IP4 Intel(R) Ethernet Controller X540-AT2       /Pci(0x2,0x0)/Pci(0x0,0x1)/MAC(2c600c945ced,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0003* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(a0369f85d164,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0004* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x0)/Pci(0x0,0x1)/MAC(a0369f85d166,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0005* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x2)/Pci(0x0,0x0)/MAC(a0369f85d0dc,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0006* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x2,0x2)/Pci(0x0,0x1)/MAC(a0369f85d0de,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0007* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x0)/MAC(a0369f85d300,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0008* UEFI: IP4 Intel(R) Ethernet Converged Network Adapter X540-T2 /Pci(0x3,0x0)/Pci(0x0,0x1)/MAC(a0369f85d302,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0009* Red Hat Enterprise Linux      HD(1,GPT,f88c1399-d2c9-48a1-a531-e904dac0f677,0x800,0x200000)/File(\EFI\REDHAT\SHIM.EFI)
Boot000A* UEFI OS       HD(1,GPT,f88c1399-d2c9-48a1-a531-e904dac0f677,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

# ls -al /boot/efi/EFI/redhat/shim*
-rwx------ 1 root root 951296 Apr 17 20:18 /boot/efi/EFI/redhat/shim.efi
-rwx------ 1 root root 951296 Apr 17 20:18 /boot/efi/EFI/redhat/shimx64.efi
-rwx------ 1 root root 944552 Apr 17 20:18 /boot/efi/EFI/redhat/shimx64-redhat.efi


> possibly running memmap from the EFI Shell.

I don't think fujitsu has an efi shell, at least I did not see an option to enter one. maybe ask the NEC guys to run this on their hardware, they should know their hardware :)

Greetings
Klaas

Comment 23 Marta Lewandowska 2023-06-02 08:52:13 UTC

Hi Klaas,
thanks for all of that... I was interested also in the size of the ESP, but default lsblk output changes, so I should've asked for something like:
# lsblk --output name,fstype,uuid,fsavail,fsuse%,mountpoints /dev/sda1

thanks again.

Comment 24 Klaas Demter 2023-06-02 08:58:51 UTC

(In reply to Marta Lewandowska from comment #23)
> Hi Klaas,
> thanks for all of that... I was interested also in the size of the ESP, but
> default lsblk output changes, so I should've asked for something like:
> # lsblk --output name,fstype,uuid,fsavail,fsuse%,mountpoints /dev/sda1
> 
> thanks again.

# lsblk --output name,fstype,uuid,fsavail,fsuse%,mountpoints /dev/sda1
lsblk: unknown column: fsavail,fsuse%,mountpoints


Keep in mind, those are rhel7 systems, so maybe test your commands on one of those :)

Comment 25 Klaas Demter 2023-06-02 09:03:23 UTC

root # df -h /boot/efi/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1      1022M   14M 1009M   2% /boot/efi

if it's just the space you worry about, it looks like this on all of them :)

Comment 26 Renaud Métrich 2023-06-02 09:51:57 UTC

Hi Klaas,

I think there is confusion. Marta is interested in knowing how large is the NVRAM, not the /boot/efi partition.
But I don't know how to collect the size.

Additionally, the idea with booting the different Shim I uploaded to the case while listing content of /sys/firmware/efi/efivars was to check for a difference, but this requires doing the operation on the same system, not different ones, for comparison.

Renaud.

Comment 27 Renaud Métrich 2023-06-02 12:24:54 UTC

Assuming we can trust the the screenshot, we hang while calling SetVariables(), since we never see line 454 in mirror_mok_db() function.

We hence have the following call stack:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
 856 EFI_STATUS import_one_mok_state(struct mok_state_variable *v,
 857                                 BOOLEAN only_first)
 858 {
 :
 926         ret = maybe_mirror_one_mok_variable(v, ret, only_first);

---

 822 static EFI_STATUS NONNULL(1)
 823 maybe_mirror_one_mok_variable(struct mok_state_variable *v,
 824                               EFI_STATUS ret, BOOLEAN only_first)
 825 {
 :
 836                 efi_status = mirror_one_mok_variable(v, only_first);

---

 504 static EFI_STATUS NONNULL(1)
 505 mirror_one_mok_variable(struct mok_state_variable *v,
 506                         BOOLEAN only_first)
 507 {
 :
 766                 efi_status = mirror_mok_db(v->rtname, (CHAR8 *)v->rtname8, v->guid,
 767                                            attrs, FullData, FullDataSize,
 768                                            only_first);

---

 313 static EFI_STATUS
 314 mirror_mok_db(CHAR16 *name, CHAR8 *name8, EFI_GUID *guid, UINT32 attrs,
 315               UINT8 *FullData, SIZE_T FullDataSize, BOOLEAN only_first)
 316 {
 :
 452                 efi_status = mirror_one_esl(namen, guid, attrs,
 453                                             esl, esd, howmany);
 454                 dprint(L"esd:0x%llx adj:0x%llx\n", esd, adj);

---

 267 static EFI_STATUS
 268 mirror_one_esl(CHAR16 *name, EFI_GUID *guid, UINT32 attrs,
 269                EFI_SIGNATURE_LIST *esl, EFI_SIGNATURE_DATA *esd,
 270                SIZE_T howmany)
 271 {
 :
 299         dprint(L"new esl:\n");
 300         dhexdumpat(var, varsz, 0);
 301 
 302         efi_status = SetVariable(name, guid, attrs, varsz, var);
 :
 310         return efi_status;
 311 }
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

What bothers me is the line number for the dhexdumpat() macro on line 300 is not matching the screenshot, since screenshot shows line 298 instead.

So 2 possibilities:
- either line number is not always correct
- or the shim binary is not the one we believe we are checking the code for

Comment 28 Renaud Métrich 2023-06-02 12:35:14 UTC

OK, the Source RPM used to build shim-x64-15.6-3.el7 is DIFFERENT than the one found on Package Browser:

- Brew one (used to build the package I suppose, https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2295859)

  SHA-256: de1bd8f759b7f6657b277d13b2098cdaf7b87dff0b10110f5db8530f6d8c50f7

- Package Browser (https://access.redhat.com/downloads/content/shim-x64/15.6-3.el7_9/x86_64/fd431d51/package)

  SHA-256: 7198d8686e6ebebbfb9c388c80899450239a9279d7a207972a6eb5133da23445

Looks like we are comparing apples and pears (sorry, french idiom).

Comment 29 Renaud Métrich 2023-06-02 12:46:38 UTC

It appears the source code is identical, which means line numbering is not always correct.
False alarm then.

Comment 30 Klaas Demter 2023-06-05 08:47:33 UTC

(In reply to Renaud Métrich from comment #26)
> Hi Klaas,
> 
> I think there is confusion. Marta is interested in knowing how large is the
> NVRAM, not the /boot/efi partition.
> But I don't know how to collect the size.
> 
> Additionally, the idea with booting the different Shim I uploaded to the
> case while listing content of /sys/firmware/efi/efivars was to check for a
> difference, but this requires doing the operation on the same system, not
> different ones, for comparison.

I can only compare that on systems that boot shim 15.6-3, if that is what you want I can list efivars on the Dell Broadwell server with different shims. But I'd guess you can do that yourself on any Broadwell system you have in your rhel testing farm :)

> 
> Renaud.

Comment 31 Renaud Métrich 2023-06-05 09:04:15 UTC

Hi Klaas,

yes the idea is to compare the outputs on exact same system between the different releases of Shim, all while in verbose mode to collect the logs.

This should rule out an issue with the UEFI NVRAM size.

And unfortunately none of our Broadwell systems I have access to fails similarly.
But most of them are used by other guys, unfortunately, so I couldn't test on many.

Renaud.

Comment 33 Renaud Métrich 2023-06-06 11:54:58 UTC

Comment 29 wrong, when diffing the bases, I unintentionally copied the ".git" directory.
I see indeed some slight changes, RHEL7 has some more stuff.

Comment 34 Klaas Demter 2023-06-06 13:01:16 UTC

I have uploaded a video from the same hardware with shim 15.6-1.el8 working and shim 15.6-3.el7 hanging.

Looking at the videos the first difference I notice:
FullDataSize 964 vs FulldDtasSze 17120

The nvram size were looking for @mlewando is that max_var_sz?
Anyways at both boots max_var_sz: A993 is shown in debug output

Comment 35 Klaas Demter 2023-06-06 13:02:54 UTC

No sorry, mav_var_sz on working boot: A5AA, notworking boot mav_var_sz: A993

Comment 36 Renaud Métrich 2023-06-06 13:48:58 UTC

Will upload the interesting videos later (currently in the customer case).

The major difference is indeed the size of MokList:

- RHEL8 shim: FullDataSize:964
- RHEL7 shim: FullDataSize:17120

This is seen as soon as Shim boots (notworking-screenshot1.png / working-screenshot1.png).
At this time, SetVariable("MokListRT") is always a success.

Then "MokListX" is mirrored, but the var is empty in both cases.

Then another diff:

- RHEL8 shim mirrors MokDBState, then SbatLevel
- RHEL7 shim mirrors SbatLevel

Later MokListTrusted and MokPolicy are mirrored, all good, but RHEL8 video skipped the FullDataSize for MokListTrusted.

The code then reads MokList again, which differs since MokList on RHEL8 is very small compared to RHEL7 one (notworking-screenshot2.png / working-screenshot2.png).
Once done, we now have:

- RHEL8 shim: FullDataSize: 1928 (twice the initial 964)
- RHEL7 shim: FullDataSize: 34240 (twice the initial 17120)

Finally MokListX is read again and grubx64/efi loads on RHEL8 (working-screenshot3.png).

NOWHERE on RHEL8 shim I see "mirror_one_esl()" being called at all.

On RHEL7 shim, I see "mirror_one_esl()" starting (notworking-screenshot3.png, remaining_sz:6600), succeeding (notworking-screenshot4.png, remaining_sz:26CE), then executing again, but we don't see the "remaining_sz" at that time, too bad).

The third round executes and hangs just after printing, probably because SetVariable() cannot succeed due to having low space.

I would expect SetVariable() to fail instead of hang, it looks like to me it's a UEFI firmware implementation bug.

Comment 37 Renaud Métrich 2023-06-06 13:50:10 UTC

Created attachment 1969301 [details]
Archive of screenshots extracted from the videos in case

Comment 38 Renaud Métrich 2023-06-06 14:11:36 UTC

Full videos (stripped from POST) available here if needed: https://people.redhat.com/rmetrich/bz2211029/

Comment 39 Marta Lewandowska 2023-06-06 15:31:42 UTC

it's not the size of the nvram, but is being set by a call to a UEFI function like this:
efi_status = RT->QueryVariableInfo(attrs, &max_storage_sz, &remaining_sz, &max_var_sz);
max_var_sz corresponds (understandably) to MaximumVariableSize.

from https://uefi.org/specs/UEFI/2.10/08_Services_Runtime_Services.html

typedef
EFI_STATUS
QueryVariableInfo (
   IN UINT32             Attributes,
   OUT UINT64            *MaximumVariableStorageSize,
   OUT UINT64            *RemainingVariableStorageSize,
   OUT UINT64            *MaximumVariableSize
  );

The QueryVariableInfo() function allows a caller to obtain the information about the maximum size of the storage space available for the EFI variables, the remaining size of the storage space available for the EFI variables and the maximum size of each individual EFI variable, associated with the attributes specified.

The MaximumVariableSize value will reflect the overhead associated with the saving of a single EFI variable with the exception of the overhead associated with the length of the string name of the EFI variable.

Comment 40 Peter Jones 2023-06-07 19:30:22 UTC

So, I think part of what's happening is that after SetVariable() fails, we're crashing printing the error because of an error in that format string:

          efi_status = SetVariable(name, guid, attrs, varsz, var);
          FreePool(var);
          if (EFI_ERROR(efi_status)) {
                  LogError(L"Couldn't create mok variable \"%s\": %r\n",
                           varsz, var, efi_status);
                  return efi_status;
          }

varsz appears to have been copied from a different error report and not belong there, and is presumably interpreted as the address that would correspond to some very low page in RAM that isn't mapped, and then it faults. As a result, we don't report the error and continue, don't mirror the variables to the Configuration Table, etc.

That still leaves the question as to why SetVariable() is failing - we're asking to create a volatile variable, i.e. it only exists in RAM.  Presumably we're over the size limit, but that makes me wonder what the justification for such a low limit might be?

Comment 42 Klaas Demter 2023-06-12 09:33:23 UTC

Created attachment 1970402 [details]
New screenshots taken with the rpm from case

Comment 43 Klaas Demter 2023-06-12 09:35:47 UTC

Attached screenshots of the hanging fujitsu servers with the latest shim rpm from case. Full video attached to case.

Comment 44 Renaud Métrich 2023-06-12 12:37:49 UTC

With adding a trace just before UEFI SetVariable() call, we can see that it's the UEFI function that hangs:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
mok.c:299:mirror_one_esl() new esl:
...
mok.c:300:mirror_one_esl() UEFI function being called...
mok.c:298:mirror_one_esl() SetVariable("MokLitRT3", ... varsz=0x229C) = Success

 :

mol.c:253:get_max_var_sz() max_var_sz:28 remaining_sz:42 max_storage_sz:10000
mok.c:429:mirror_mok_db() max_var_sz - name: 12
mok.c:281:mirror_one_esl() Trying to add 98 signatures to "MokListRT4" of size 30
mok.c:299:mirror_one_esl() new esl:
...
mok.c:300:mirror_one_esl() UEFI function being called...
--> HANG
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Clearly the fact it lacks space should return an error instead of hanging.
So, to me, there is a bug in the UEFI firmware implementation provided by Fujitsu.

The workaround is likely to not try since we know there won't be enough space, but this should be considered a hack and Shim should be able to rely on efi_status and not do pre-checks.

Comment 45 Klaas Demter 2023-06-14 10:17:35 UTC

It seems this also affects Fujitsu PRIMEQUEST 2800B2/3

Comment 53 raravind 2023-07-06 17:21:23 UTC

Jared,

We are engaging Fujitsu in this matter and they are testing the same in their servers.It is more likely firmware issue.
https://bugzilla.redhat.com/show_bug.cgi?id=2209291#c16

Comment 57 comsec 2023-08-09 20:17:40 UTC

It does happen also with shim-x64-15.6-1.el8 on RHEL8 (included on 8.7 and 8.8 that fail to install) with a Fujitsu Primergy RX300 S8 and  Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: FUJITSU // American Megatrends Inc.
        Version: V4.6.5.4 R1.22.0 for D2939-B1x
        Release Date: 05/17/2019
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 8192 kB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Print screen service is supported (int 5h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 1.22

Comment 58 comsec 2023-08-09 20:39:18 UTC

rhel 8.6 with shim-x64-15.5-2.el8.x86_64 same install problem
keep in mind that uefi is enabled but secure boot is disabled

(In reply to comsec from comment #57)
> It does happen also with shim-x64-15.6-1.el8 on RHEL8 (included on 8.7 and
> 8.8 that fail to install) with a Fujitsu Primergy RX300 S8 and  Intel(R)
> Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> 
> Handle 0x0000, DMI type 0, 24 bytes
> BIOS Information
>         Vendor: FUJITSU // American Megatrends Inc.
>         Version: V4.6.5.4 R1.22.0 for D2939-B1x
>         Release Date: 05/17/2019
>         Address: 0xF0000
>         Runtime Size: 64 kB
>         ROM Size: 8192 kB
>         Characteristics:
>                 PCI is supported
>                 BIOS is upgradeable
>                 BIOS shadowing is allowed
>                 Boot from CD is supported
>                 Selectable boot is supported
>                 EDD is supported
>                 Print screen service is supported (int 5h)
>                 Serial services are supported (int 14h)
>                 Printer services are supported (int 17h)
>                 ACPI is supported
>                 USB legacy is supported
>                 BIOS boot specification is supported
>                 Targeted content distribution is supported
>                 UEFI is supported
>         BIOS Revision: 1.22

Comment 59 Klaas Demter 2023-08-09 21:16:28 UTC

 @raravind I am guessing the comment you asked me a question in is private :) Please comment publicly then I'll happily provide all input I can offer.

Comment 60 raravind 2023-08-09 21:30:25 UTC

Could you please try the patch mentioned in comment#55?
In other words - Please try the patch - https://github.com/rhboot/shim/commit/dbbe3c84bd0e7683d4b81c1794a112a6853b80ee to shim-15.6-3.el7.src.rpm to see if it is working for you as mentioned in 
https://bugzilla.redhat.com/show_bug.cgi?id=2220848#c35

By putting a needinfo I meant this :)

Comment 61 comsec 2023-08-10 14:33:35 UTC

Rhel 8.5 finally works and seems to install just fine and it has shim-x64-15.4-2.el8_1.x86_64

So this bug on red hat 8 was introduced in shim 15.5

Comment 62 Marta Lewandowska 2023-08-10 14:43:45 UTC

@comsec are you able to try the patch mentioned in comment#60 on any of the shims that you have found not to work? We unfortunately don't have the hardware to reproduce or test this.

Comment 63 comsec 2023-08-10 14:52:17 UTC

(In reply to Marta Lewandowska from comment #62)
> @comsec are you able to try the patch mentioned in comment#60 on
> any of the shims that you have found not to work? We unfortunately don't
> have the hardware to reproduce or test this.

Unfortunately these systems are being prepared for production, so installing development software and compiling it's not possible now.
If you're able to provide a rhel8 binary rpm it will be much more simpler to give it a try.

Comment 64 comsec 2023-08-10 14:54:15 UTC

In the meantime better modify products impacted by this bug to rhel7 and rhel8, that will totally change perspective I suppose.

Comment 65 comsec 2023-08-10 15:06:25 UTC

Created attachment 1982817 [details]
boot error message on RHEL8

Comment 66 comsec 2023-08-10 15:25:38 UTC

And judjing by correspondent Fedora Bug 2113005 it seems to affect possibly all UEFI AMI BIOS from a certain time period
So potentially mainboards from Asus, MSI, Supermicro, Lenovo, ecc just to name a few

Comment 67 comsec 2023-08-10 16:09:54 UTC

Not by chance last Fedora testday on shim was with 15.4, nothing after that

https://fedoraproject.org/wiki/Test_Day:2021-04-12_Grub_and_Shim_Test_Day

Comment 68 Klaas Demter 2023-08-10 23:25:15 UTC

Hi @comsec 
I think you should create a new bug, your report does not seem to fit this one. With this bug you'd just see a blank screen when not using debug, and with debug something like https://bugzilla.redhat.com/show_bug.cgi?id=2211029#c3

Greetings
Klaas

Comment 69 Marta Lewandowska 2023-08-11 10:14:32 UTC

(In reply to comsec from comment #63)
> Unfortunately these systems are being prepared for production, so installing
> development software and compiling it's not possible now.
> If you're able to provide a rhel8 binary rpm it will be much more simpler to
> give it a try.

I agree with comment#68 that this is a different issue, so please do file a new bug with some more information and logs, including debug and/or verbose options, if possible.

Also, the fedora bug that you mentioned is for shim-x64-15.6-2, which isn't in RHEL.

Comment 70 Klaas Demter 2023-08-28 14:34:01 UTC

(In reply to raravind from comment #60)
> Could you please try the patch mentioned in comment#55?

This comment is private and I can't see it

> In other words - Please try the patch -
> https://github.com/rhboot/shim/commit/
> dbbe3c84bd0e7683d4b81c1794a112a6853b80ee to shim-15.6-3.el7.src.rpm to see
> if it is working for you as mentioned in 
> https://bugzilla.redhat.com/show_bug.cgi?id=2220848#c35

That bug is private as well, I have no access to it

> 
> By putting a needinfo I meant this :)

I'll get support to supply a rpm/binary in case 03525053 and then I'll test it. Will report back with results as soon as possible.

Comment 71 Marta Lewandowska 2023-08-30 07:51:30 UTC

(In reply to Klaas Demter from comment #70)
> (In reply to raravind from comment #60)
> > Could you please try the patch mentioned in comment#55?
> 
> This comment is private and I can't see it

https://github.com/rhboot/shim/commit/dbbe3c84bd0e7683d4b81c1794a112a6853b80ee

> > In other words - Please try the patch -
> > https://github.com/rhboot/shim/commit/
> > dbbe3c84bd0e7683d4b81c1794a112a6853b80ee to shim-15.6-3.el7.src.rpm to see
> > if it is working for you as mentioned in 
> > Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2220848#c35
> 
> That bug is private as well, I have no access to it

I've added you to that bug.

> > 
> > By putting a needinfo I meant this :)
> 
> I'll get support to supply a rpm/binary in case 03525053 and then I'll test
> it. Will report back with results as soon as possible.

if you need rpms built, let us know. :)

Comment 76 Klaas Demter 2023-08-31 11:42:20 UTC

(In reply to Marta Lewandowska from comment #71)
> (In reply to Klaas Demter from comment #70)
> > (In reply to raravind from comment #60)
> > > Could you please try the patch mentioned in comment#55?
> > 
> > This comment is private and I can't see it
> 
> https://github.com/rhboot/shim/commit/
> dbbe3c84bd0e7683d4b81c1794a112a6853b80ee
> 
> > > In other words - Please try the patch -
> > > https://github.com/rhboot/shim/commit/
> > > dbbe3c84bd0e7683d4b81c1794a112a6853b80ee to shim-15.6-3.el7.src.rpm to see
> > > if it is working for you as mentioned in 
> > > Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2220848#c35
> > 
> > That bug is private as well, I have no access to it
> 
> I've added you to that bug.

I still can't access that bug

> 
> > > 
> > > By putting a needinfo I meant this :)
> > 
> > I'll get support to supply a rpm/binary in case 03525053 and then I'll test
> > it. Will report back with results as soon as possible.
> 
> if you need rpms built, let us know. :)

Thanks I got the rpm. I can say that shim-unsigned-x64-15.6-9.el7.x86_64 seems to boot again.

Comment 77 Marta Lewandowska 2023-09-01 12:04:39 UTC

(In reply to Klaas Demter from comment #76)
> (In reply to Marta Lewandowska from comment #71)
> > (In reply to Klaas Demter from comment #70)
> > > (In reply to raravind from comment #60)
> > > > Could you please try the patch mentioned in comment#55?
> > > 
> > > This comment is private and I can't see it
> > 
> > https://github.com/rhboot/shim/commit/
> > dbbe3c84bd0e7683d4b81c1794a112a6853b80ee
> > 
> > > > In other words - Please try the patch -
> > > > https://github.com/rhboot/shim/commit/
> > > > dbbe3c84bd0e7683d4b81c1794a112a6853b80ee to shim-15.6-3.el7.src.rpm to see
> > > > if it is working for you as mentioned in 
> > > > Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2220848#c35
> > > 
> > > That bug is private as well, I have no access to it
> > 
> > I've added you to that bug.
> 
> I still can't access that bug

Yeah, sorry. I added you without asking, which was a mistake on my part. If you would like to be able to see it, I will ask Fujitsu if they are ok with that.

> > 
> > > > 
> > > > By putting a needinfo I meant this :)
> > > 
> > > I'll get support to supply a rpm/binary in case 03525053 and then I'll test
> > > it. Will report back with results as soon as possible.
> > 
> > if you need rpms built, let us know. :)
> 
> Thanks I got the rpm. I can say that shim-unsigned-x64-15.6-9.el7.x86_64
> seems to boot again.

that's good news. thanks for letting us know.

Comment 78 raravind 2023-09-04 08:43:18 UTC

Klass,Thank you for your patience and co-operation :). Could we close this bz if the solution works for you?

Comment 79 Klaas Demter 2023-09-04 09:26:34 UTC

(In reply to raravind from comment #78)
> Klass,Thank you for your patience and co-operation :). Could we close this
> bz if the solution works for you?

I am guessing you want to close this bz via errata once a new shim-x64 release is public :)

Comment 80 comsec 2023-09-04 10:56:42 UTC

Our exact issue seems already open as Bug 2113005 but on Fedora only

This at first glance seemed the same but probably is not.

Anyway the issue is with whatever shim starting from 15.5

(In reply to Marta Lewandowska from comment #69)
> (In reply to comsec from comment #63)
> > Unfortunately these systems are being prepared for production, so installing
> > development software and compiling it's not possible now.
> > If you're able to provide a rhel8 binary rpm it will be much more simpler to
> > give it a try.
> 
> I agree with comment#68 that this is a different issue, so please do file a
> new bug with some more information and logs, including debug and/or verbose
> options, if possible.
> 
> Also, the fedora bug that you mentioned is for shim-x64-15.6-2, which isn't
> in RHEL.

Comment 81 Klaas Demter 2023-09-04 11:59:59 UTC

(In reply to comsec from comment #80)
> Our exact issue seems already open as Bug 2113005 but on Fedora only
> 
> This at first glance seemed the same but probably is not.
> 
> Anyway the issue is with whatever shim starting from 15.5
> 
> (In reply to Marta Lewandowska from comment #69)
> > (In reply to comsec from comment #63)
> > > Unfortunately these systems are being prepared for production, so installing
> > > development software and compiling it's not possible now.
> > > If you're able to provide a rhel8 binary rpm it will be much more simpler to
> > > give it a try.
> > 
> > I agree with comment#68 that this is a different issue, so please do file a
> > new bug with some more information and logs, including debug and/or verbose
> > options, if possible.
> > 
> > Also, the fedora bug that you mentioned is for shim-x64-15.6-2, which isn't
> > in RHEL.

You need to open a new issue for your problem as stated in https://bugzilla.redhat.com/show_bug.cgi?id=2211029#c68 and https://bugzilla.redhat.com/show_bug.cgi?id=2211029#c69

The description of your problem does not fit the issue in this bugzilla entry.

Comment 83 RHEL Program Management 2023-09-16 19:06:15 UTC

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 84 RHEL Program Management 2023-09-16 20:07:13 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.