It appears OpenQA tests are also failing: https://openqa.fedoraproject.org/tests/3156457#step/_boot_to_anaconda/3

aarch64 only, right? not x86?

correct. All other architectures are passing tests.

Hi Dusty, was it working fine with grub2-2.12-17.fc42 and the same EDK2 version (edk2-20241117-5.fc41) ?

Any chance this firmware has a way to turn on ConErr or edk2 debug output?  I can't even tell if it ran grub or which PC addresses might be grub vs firmware from this output.

The openQA aarch64 worker hosts actually still have the old edk2 - 20240813 - and they are still hitting this. It started with Fedora-Rawhide-20250110.n.0 , which is exactly when grub went from 2.12-15.fc42 to 2.12-18.fc42 .

note that -16 and -17 were never built, so we went straight from -15 to -18. the changelog shows:

  * Wed Nov 27 2024 Marta Lewandowska <mlewando> - 2.12-16
  - 99-grub-mkconfig.install: Disable BLS and run grub2-mkconfig when GRUB_ENABLE_BLSCFG is disable
  - Resolves: #2325960

  * Wed Nov 27 2024 Marta Lewandowska <mlewando> - 2.12-17
  - 99-grub-mkconfig.install: on PPC systems, remove petiboot's version checks

  * Thu Jan 09 2025 Nicolas Frayer <nfrayer> - 2.12-18
  - fs/xfs: fix large extent counters incompat feature support

(In reply to Nicolas Frayer from comment #4)
> Hi Dusty, was it working fine with grub2-2.12-17.fc42 and the same EDK2
> version (edk2-20241117-5.fc41) ?

As Adam mentioned the -17 was never built. 


(In reply to Peter Jones from comment #5)
> Any chance this firmware has a way to turn on ConErr or edk2 debug output? 
> I can't even tell if it ran grub or which PC addresses might be grub vs
> firmware from this output.


We're just using virt tools (i.e. QEMU) in a test framework so I assume the answer is yes. Here is a link to an ISO showing the problem if you'd like to poke:

https://dustymabe.fedorapeople.org/fedora-coreos-42.20250121.dev.0-live-iso.aarch64.iso

(In reply to Peter Jones from comment #5)
> Any chance this firmware has a way to turn on ConErr or edk2 debug output? 

Yes.  It's a compile time option, so there are two images (QEMU_EFI-pflash + QEMU_EFI-silent-pflash).  Use the one not named 'silent' to get the firmware log on the serial console.

(In reply to Peter Jones from comment #5)
> I can't even tell if it ran grub or which PC addresses might be grub vs
> firmware from this output.

It's grub.  Maybe shim.

(In reply to Dusty Mabe from comment #0)
> Synchronous Exception at 0x000000013DA8A0E8
> PC 0x00013DA8A0E8
> PC 0x00013DA8A080
> PC 0x00013DA8A32C
> PC 0x00013DA8A420
> PC 0x00013DF7524C
> PC 0x00013DF779E4
> PC 0x00013DF8068C
> PC 0x00013DF811A8
> PC 0x00013E78E574
> PC 0x00013E78E624
> PC 0x00013E78F9A8
> PC 0x00013E78C030
> PC 0x00004769B2C0 (0x000047694000+0x000072C0) [ 1] DxeCore.dll
> PC 0x00013F14AECC (0x00013F144000+0x00006ECC) [ 2] BdsDxe.dll
> PC 0x00013F14CCF0 (0x00013F144000+0x00008CF0) [ 2] BdsDxe.dll
> PC 0x00004769DFF8 (0x000047694000+0x00009FF8) [ 3] DxeCore.dll

This is a simple stack trace.  Anything the edk2 exception handler can't map to firmware modules is not firmware code.

You're right, Gerd, we're in grub.

kern/device.c:37:device: opening device cd0
kern/disk.c:196:disk: Opening `cd0'...
disk/efi/efidisk.c:495:efidisk: opening cd0
disk/efi/efidisk.c:524:efidisk: m = 0x13eaaa510, last block = 7767a, block size
= 800, io align = 0
disk/efi/efidisk.c:542:efidisk: opening cd0 succeeded
kern/disk.c:288:disk: Opening `cd0' succeeded.
kern/disk.c:196:disk: Opening `cd0'...
disk/efi/efidisk.c:495:efidisk: opening cd0
disk/efi/efidisk.c:524:efidisk: m = 0x13eaaa510, last block = 7767a, block size
= 800, io align = 0
disk/efi/efidisk.c:542:efidisk: opening cd0 succeeded
kern/disk.c:288:disk: Opening `cd0' succeeded.


Synchronous Exception at 0x000000013D5A00E8
...

And this is at 2.12-15 (see comment#7), so now to revert more patches...

I don't know what the problem is yet, but it looks like grub can't properly open the boot.iso, so I can only guess that something has changed in the way that it's being created? I've taken grub back to 2.12-9 (on a rawhide builder), which was originally built in the beginning of October, and I see that this test was working as recently as 10 Jan ... OTOH I can boot the f-41 iso without any issues.

I am building my boot isos with lorax and then genisoimage on a rawhide VM and they are clearly failing just like the official ones...
[root@fedora ~]# rpm -qa genisoimage lorax
lorax-42.4-1.fc42.aarch64
genisoimage-1.1.11-56.fc41.aarch64

And I'm on an f-41 hypervisor with
[root@ampere-hr330a-13 ~]# rpm -qa edk2-aarch64
edk2-aarch64-20241117-5.fc41.noarch

I don't believe we've changed anything about how we build netinst and DVD ISOs, no. On openQA the failure is happening with the Server DVD ISO and I don't think we intentionally changed that build process. We have been moving a lot of stuff to Kiwi, but not those ISOs. As I said, the failures started exactly on the compose where grub2-1:2.12-18.fc42 landed - Fedora-Rawhide-20250110.n.0 . There were no changes to pungi-fedora between those composes AFAICS. No packages changed on the compose host in that time frame (there was a change on Jan 6 and then Jan 13). Lorax hasn't changed since November...

To further support the argument that it's likely grub and not the way we build things:

1. It happened in Fedora CoreOS and Fedora Server (in openQA) at the same time
2. When we reverted the change in Fedora CoreOS [1] we are now able to build and test aarch64 just fine. This is with everything else staying the same except GRUB getting reverted.

[1] https://github.com/coreos/fedora-coreos-config/pull/3330

Oh, it's great that you tested that, Dusty, I was about to do the same thing, saves me the time :D

I just ran into this with Fedora-Server-dvd-aarch64-Rawhide-20250123.n.0.iso on both a F40 and a F41 qemu too. There have been some similar UTM reports from not long ago about CDROMs triggering similar behavior with fedora. SO it looks like a NULL pointer deference from the FAR, presumably something in grubs ISO handling? I'm building a more verbose edk2/grub lets see if that helps.

Sorta as expected grub2-efi-aa64-2.12.18 and grub2-efi-aa64-cdboot-2.12.18 FC42 boot from a disk.

Since I'm having a build issue, and am about to quit for the night, it might be useful to drop `19dcf16 aarch64/macros: Build gnulib with -mbranch-protection=standard` and see if that fixes it, since the behavior changed a bit with gcc15.

After reading your comment yesterday, Dusty, I was feeling a little stupid and confused, but things are making a little bit more sense... maybe...

I built a DVD with f-41, and then tried grub 2.12-15.fc42 and 2.12-18.fc42, and -15 booted while -18 did not, as per Dusty's findings.

So then on an f-41 builder, I started from grub 2.12-15.fc42 and then built -16, -17, -18. I created a DVD with f-41 along with each built grub, and all of them booted.

Previously, what I reported in comment#12, was all built on a rawhide builder, and all of those were hitting the exception regardless of the included patches. So this must be something with builddeps, changes in libraries, or something along those lines.

Nicolas built 2.12-21.fc42 without branch protection that Jeremy mentioned, but that also hit the exception.

I'm gonna keep playing with this.

I can't seem to find valid debuginfo's for the shipped grub2-efi image. So if one opens the resulting image like:
gdb grubaa64.efi 
Reading symbols from ./usr/lib/debug/.dwz/grub2-2.12-20.fc42.aarch64...
(gdb) disassemble 0x2c11c0,+300
...
0x00000000002c12a0:  ldr     x0, [x0]
0x00000000002c12a4:  ldr     x0, [x0, #40]
0x00000000002c12a8:  ldr     x0, [x0, #48]<--- crash access 0x030
0x00000000002c12ac:  ldr     x2, [x0, #16]


This is based on the running memory offset/instructions offset into a standalone gdb image. I'm not sure I trust the edk2 stack unwinding in grub though, but that address matches the 0x30 FAR offset and X0=0 in the backtrace I see. So its likely the faulting instruction, the only question at the moment is why the instruction immediately before it didn't fault too. I waa trying to rebuilt an identical grub the other day to use its debug symbols but it wasn't helpful. If someone shoots me a busted image with debug symbols in the grub.efi (or the kernel core with symbols) I can use that to lookup the function name.

ok, please disregard all of my other theories above ;) I found the thing that is causing this, although I don't know why it's causing this...
I did a diff between the working and non-working grubs, and I found a couple of surprises, based on the changelog:
[mlewando@loki r10]$ diff -qr 15/ 18/
Only in 18/: 0284-fs-xfs-Fix-large-extent-counters-incompat-feature-su.patch
Files 15/20-grub.install and 18/20-grub.install differ                      <------
Files 15/99-grub-mkconfig.install and 18/99-grub-mkconfig.install differ
Only in 15/: grub2-2.12-15.fc42.src.rpm
Only in 18/: grub2-2.12-18.fc42.src.rpm
Files 15/grub2.spec and 18/grub2.spec differ
Files 15/grub.macros and 18/grub.macros differ                              <------
Files 15/grub.patches and 18/grub.patches differ

It turns out that adding the (new) bli module to the global modules is the culprit. I looked briefly at the bli code, but haven't identified the issue there. I don't know why this isn't an issue on x86 or why it's only an issue on aarch booting from cd/dvd...
[mlewando@loki r10]$ diff -ruN 15/grub.macros 18/grub.macros 
--- 15/grub.macros	2024-11-21 01:00:00.000000000 +0100
+++ 18/grub.macros	2025-01-09 01:00:00.000000000 +0100
@@ -140,7 +140,7 @@
 %{?with_efi_only:%global without_efi_only 1}
 
 %ifarch %{efi_arch}
-%global efi_modules " efi_netfs efifwsetup efinet lsefi lsefimmap connectefi "
+%global efi_modules " efi_netfs efifwsetup efinet lsefi lsefimmap connectefi bli "
 %endif

Anyway, Jeremy, if you want a debug grub, I can build you one. I guess you want the broken version, -18?

Every time I debug grub i get PTSD.. it shouldn't be this hard to generate debug images.

So, your right it is the bli module, its in get_part_uuid, line 59:

   {
      status = grub_error (grub_errno, N_("cannot open disk: %s"), device_name);
      grub_device_close (device);
      return status;
    }

  if (grub_strcmp (device->disk->partition->partmap->name, "gpt") != 0)
    {
      status = grub_error (GRUB_ERR_BAD_PART_TABLE,
			   N_("this is not a GPT partition table: %s"), device_name);
      goto fail;
    }


Where I think partmap is null if i'm following the assembly dereferences correctly, since at this point the idea of successfully manipulating the build into generating a listing.

  dc:   f94057e0        ldr     x0, [sp, #168] <--- device
  e0:   f9400000        ldr     x0, [x0]       <--- disk
  e4:   f9401400        ldr     x0, [x0, #40]  <--- partition
  e8:   f9401800        ldr     x0, [x0, #48]  <--- partmap (death!)
  ec:   f9400802        ldr     x2, [x0, #16]  <---- name (parm #1 to grub_strcmp after it gets moved to x0 below, this code needs the optimizer...)
  f0:   90000000        adrp    x0, 0 <get_part_uuid> 
  f4:   91000000        add     x0, x0, #0x0
  f8:   f9400001        ldr     x1, [x0]
  fc:   aa0203e0        mov     x0, x2
 100:   94000000        bl      0 <grub_strcmp>

Right, so if the device name doesn't have a partition element the device->disk->partition doesn't get filled out by grub_disk_open() so at a minimum adding a

if (device->disk->partition && grub_strcmp(device->disk->partition-partmap->name, "gpt") != 0) 

is required.

Dope! Upstream commit: 

9537f4403 commands/bli: Fix crash in get_part_uuid()

From the bug:


    4. When booting from a CD-ROM, the ESP is a VFAT image indexed by the El
       Torito boot catalog. The boot device is set to (cd0), corresponding
       to the CD-ROM image mounted as an ISO 9660 filesystem.

So, this is probably because the x86 qemu/etc simply isn't crashing on zero page access.

> So, this is probably because the x86 qemu/etc simply isn't crashing on zero
> page access.

Correct.  That is going to change soon though, see
https://fedoraproject.org/wiki/Changes/Edk2Security

Nice sleuthing Jeremy! I can confirm that https://git.savannah.gnu.org/cgit/grub.git/commit/?id=9537f4403dd836d5a8a1c4e57b165837fc7239cf fixes this issue on aarch64.

According to git [1] this should be taken care of with grub2-2.12-21.fc42 

[1] https://src.fedoraproject.org/rpms/grub2/c/20db2b22d863543f0f31b4c54cc7e65923473f62?branch=rawhide