Created attachment 2121510 [details]
boot screen photo

Hi Mikhail,

I am quite surprise for this issue: we had a 'test day' specially for the this change (well, there are two changes really, [50] and [49]) as you can see here [1]. What we really test on the test day was [49].

Before going into the debugging, can you please go ahead and do your testing asked in [1]? Perhaps you are the unlucky one having old firmware which may be affected by this change. Make sure you add your results onto [2] or directly on this ticket.

(btw, We may close this ticket and instead work on https://bugzilla.redhat.com/show_bug.cgi?id=2263643 , the long standing ticket for this issue, but perhaps this is a different issue)

[1] https://fedoraproject.org/wiki/Test_Day:2025-12-15_GRUB_out_of_memory_verification
[2] https://testdays.fedoraproject.org/testday/11

[50] https://src.fedoraproject.org/rpms/grub2/c/85de859da1331046d29ffda9f1aac33d93da8457?branch=rawhide
[49] https://src.fedoraproject.org/rpms/grub2/c/16cb5b303d06825e8b36b5ca5e195e811d047bd1?branch=rawhide

Mikhail, from the grub prompt, could you also please `echo $root` and `echo $prefix` and perhaps `ls` and post the output of those commands? thanks.
Do you see an error before it fails?

Created attachment 2121547 [details]
Photo of results `echo $root`, `echo $prefix` and `ls`

Hi Leo, Marta!

I joined the `Test Day` and my system successfully booted from `05673235-Fedora-Workstation-Live-x86_64-139891207.iso`.

I also added my results to [2]:

- ASUS ROG STRIX B650E-I GAMING WIFI: https://pastebin.com/LyEgRq0q
- ASROCK B650I Lightning WiFi: https://pastebin.com/rjPJMF8f

> Mikhail, from the grub prompt, could you also please echo $root and echo $prefix and perhaps ls and post the output of those commands?

See the attached photo. In short:

- echo $root -> hd1
- echo $prefix -> (hd1)/EFI/fedora
- ls prints repeated errors:
.../grub-core/disk/efi/efidisk.c:grub_efidisk_read:625: failure reading sector ... from 'hd0'/'hd1'

> Do you see an error before it fails?

I did not notice any error message before it drops to the grub> prompt. The errors above only appear when running ls at the GRUB prompt.

Created attachment 2121601 [details]
Photo of results `echo $root`, `echo $prefix` and `ls`  - second computer

Small update: I can reproduce the exact same failure on a second machine (ASROCK B650I Lightning WiFi) as well. This looks like a general regression rather than a single-system quirk.

Thanks Mikhail for quickly showing us log. 

We do not understand two things: 

1- why on the test day the fix work fine and not this time? 
2- why $prefix has that value which is apparently wrong? 

we believe the partition table being read is wrong so ls is showing those sector errors and showing no partitions.

This is the action plan:

Please start again with a bootable system (grub2-2.12-48), then update first to the -49 scratch (see below)
and boot. Does it boot? it is expected to boot as the test day basically contained this change.
Then try the -50 scratch, update and reboot. Also, before rebooting after an update, make sure you enable the grub debugging
with (we are mostly interested in the log's tail but if you can get more log, better)

  grub2-editenv - set debug=all,-lexer,-scripting

this is done from a running fedora system, not the grub prompt.

Two scratch builds (non signed builds so disable secure boot from the firmware settings, in theory SB is not playing a role here),
- 2.12-49 (OOM Fix)  https://koji.fedoraproject.org/koji/taskinfo?taskID=140906146
- 2.12-50 without the OOM fix (2.12-49) but the -50 change included https://koji.fedoraproject.org/koji/taskinfo?taskID=140906368

Created attachment 2121660 [details]
frame with error message

Hi Leo, Marta,

Update on your action plan:

- Starting from a bootable state with grub2-2.12-48, upgrading to the 2.12-49 scratch build (task 140906146) is already enough to reproduce the failure.
- After installing grub2*-1:2.12-49.fc44 and rebooting, the system drops to the `grub>` prompt again (same behavior as with 2.12-50).

I enabled GRUB debugging as requested:
  grub2-editenv - set debug=all,-lexer,-scripting
but it did not add much in practice. The only debug line I managed to capture on video is in the attached screenshot (frame_000121.jpg). My phone can only record at 60 fps, so most of the output scrolls by too fast.

The captured line is:
  .../grub-core/disk/efi/efidisk.c:grub_efidisk_read:625: failure reading ...

So the regression seems to start with the 2.12-49 build already.

I'm having the same exact same issue. I actually tried to downgrade my grub2 packages after chrooting in to my install but it says  I already have the lowest version of the package installed so that doesn't appear to be any option for me.

The only thing I can really add is that I was trying to work through the issue with ChatGPT yesterday and at some point after running `insmod btrfs`, `ls` suddenly worked without throwing "failure reading sector 0x0" errors. At that point I was able to point it to the config which brought up the bootloader and I was able to get into my Rawhide install. Within the install I decided to run another 'dnf reinstall grub2*` just to see if something different happened but when I rebooted I wasn't able to get back in. `insmod btrfs` didn't made ls work anymore. Running `insmod nvme` threw up a "failure reading sector 0x0` again but for only one of my drives. On subsequent reboots, that same command produced no errors which makes me think `insmod btrfs` didn't do anything. So whatever is wrong isn't even consistent between boots.

> I already have the lowest version of the package installed so that doesn't appear to be any option for me.

If `dnf downgrade grub2*` says “lowest version”, it just means older grub2 is no longer available in your enabled repos.

You can still install an older build directly from Koji:

1) Install koji client:
   sudo dnf install -y koji

2) Download the older Fedora 44 build (example: grub2-2.12-48.fc44):
   mkdir -p ~/koji/grub2-2.12-48 && cd ~/koji/grub2-2.12-48
   koji download-build --arch=x86_64 --arch=noarch grub2-2.12-48.fc44

3) Install/downgrade from the local RPMs:
   sudo dnf downgrade --disablerepo='*' ./*.rpm
   # (or: sudo dnf install --allowerasing --disablerepo='*' ./*.rpm)

Then reboot.

Thank you! I didn't know how to find the older packages to install locally or I would have tried earlier. That worked.

Obviously now that I have a way to return to a working state, I can go back to the broken package if you could use additional testing to figure out what the actual issue is.

I can confirm I was impacted by this too.

Upgrade  grub2-common-1:2.12-50.fc44.noarch                        Dependency      rawhide
  Upgrade  grub2-tools-minimal-1:2.12-50.fc44.x86_64                 Dependency      rawhide
  Upgrade  grub2-tools-extra-1:2.12-50.fc44.x86_64                   Group           rawhide
  Upgrade  grub2-tools-efi-1:2.12-50.fc44.x86_64                     Group           rawhide
  Upgrade  grub2-tools-1:2.12-50.fc44.x86_64                         Group           rawhide
  Upgrade  grub2-pc-modules-1:2.12-50.fc44.noarch                    Dependency      rawhide
  Upgrade  grub2-pc-1:2.12-50.fc44.x86_64                            Group           rawhide
  Upgrade  grub2-efi-x64-modules-1:2.12-50.fc44.noarch               User            rawhide
  Upgrade  grub2-efi-x64-cdboot-1:2.12-50.fc44.x86_64                Group           rawhide
  Upgrade  grub2-efi-x64-1:2.12-50.fc44.x86_64                       Group           rawhide
  Upgrade  grub2-efi-ia32-modules-1:2.12-50.fc44.noarch              Dependency      rawhide
  Upgrade  grub2-efi-ia32-cdboot-1:2.12-50.fc44.x86_64               Group           rawhide
  Upgrade  grub2-efi-ia32-1:2.12-50.fc44.x86_64                      Group           rawhide

This broke booting on my system, which is a fairly standard installation on a single disk, BTRFS, no encryption:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
zram0       251:0    0     8G  0 disk [SWAP]
nvme0n1     259:0    0 953.9G  0 disk 
├─nvme0n1p1 259:1    0   600M  0 part /boot/efi
├─nvme0n1p2 259:2    0     1G  0 part /boot
└─nvme0n1p3 259:3    0 952.3G  0 part /home
                                      /

I've filed a request to untag the package from Rawhide for now - https://forge.fedoraproject.org/releng/tickets/issues/13157 . Leo will look into this urgently in the coming week. Look out for any info requests.

(In reply to Mark Grgurev from comment #10)
> Thank you! I didn't know how to find the older packages to install locally
> or I would have tried earlier. That worked.
> 
> Obviously now that I have a way to return to a working state, I can go back
> to the broken package if you could use additional testing to figure out what
> the actual issue is.

Hi Mark, I'm glad you were able to get your system back up (thanks Mikhail!) and thank you for volunteering to help test in the future. Could you please do an `lsblk -f` and paste the output so that we know something about your system? I guess you are also using btrfs. Is your /boot btrfs as well?
thanks.

Following Marta request but being a bit more general of the info we believe would be useful, please do the following 

Assuming you are on a running rawhide systems, enable grub debug and upgrade to 2.12-50 with the following

$ sudo dnf install -y koji
$ grub2-editenv - set debug=all,-lexer,-scripting
$ grub2-editenv - set pager=1
$ mkdir grub2-2.12-50
$ cd grub2-2.12-50
$ koji download-build --arch=x86_64 --arch=noarch grub2-2.12-50.fc44
$ sudo dnf install *.rpm
$ lsblk -f # place it on a pastebin, never expire
$ sudo reboot   # in the same pastebin as above, place any boot error seeing
$ #optional, just in case your system boots fine and you want to back to stable 'sudo dnf downgrade grub2\*'

Created attachment 2121958 [details]
video demonstration

Following your request, I installed grub2-2.12-50.fc44 from Koji with GRUB debug enabled (debug=all,-lexer,-scripting, pager=1) and reproduced the issue again.

> $ lsblk -f # place it on a pastebin, never expire

root@primary-ws ~# lsblk -f
NAME        FSTYPE FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
zram0       swap   1     zram0    1868a0d0-ac55-401b-b589-f856034c9dae                [SWAP]
nvme1n1                                                                               
├─nvme1n1p1 vfat   FAT16          4307-D37D                             469.9M     4% /boot/efi
├─nvme1n1p2 swap   1              49251da1-a5cf-49ab-af4e-6dff17173e89                [SWAP]
└─nvme1n1p3 btrfs        btrfs.37 b4996eda-3a45-4f45-ad59-be0bac273396    194G    37% /
nvme0n1     btrfs                 46ed41f1-2512-4e13-a502-4add0e26a341    5.6T    80% /home
root@primary-ws ~# 


> $ sudo reboot   # in the same pastebin as above, place any boot error seeing

No error messages are shown before it drops to the grub> prompt. Please see the attached video (output_vbr_6.mp4).

(In reply to Mikhail from comment #15)
> Created attachment 2121958 [details]
> video demonstration
> 
> Following your request, I installed grub2-2.12-50.fc44 from Koji with GRUB
> debug enabled (debug=all,-lexer,-scripting, pager=1) and reproduced the
> issue again.

how did you enable the logging? using grub2-editenv? In theory a LOT of msgs 
should be printed.

> 
> > $ lsblk -f # place it on a pastebin, never expire
> 
> root@primary-ws ~# lsblk -f
> NAME        FSTYPE FSVER LABEL    UUID                                
> FSAVAIL FSUSE% MOUNTPOINTS
> zram0       swap   1     zram0    1868a0d0-ac55-401b-b589-f856034c9dae      
> [SWAP]
> nvme1n1                                                                     
> 
> ├─nvme1n1p1 vfat   FAT16          4307-D37D                            
> 469.9M     4% /boot/efi
> ├─nvme1n1p2 swap   1              49251da1-a5cf-49ab-af4e-6dff17173e89      
> [SWAP]
> └─nvme1n1p3 btrfs        btrfs.37 b4996eda-3a45-4f45-ad59-be0bac273396   
> 194G    37% /
> nvme0n1     btrfs                 46ed41f1-2512-4e13-a502-4add0e26a341   
> 5.6T    80% /home
> root@primary-ws ~# 
> 
> 
> > $ sudo reboot   # in the same pastebin as above, place any boot error seeing
> 
> No error messages are shown before it drops to the grub> prompt. Please see
> the attached video (output_vbr_6.mp4).

no debug msgs shown which means that either is failing quite soon or debug is not properly
enable for some reason. 

BTW, I tried the same partition setup (btrfs on / and no /boot partition) but on VM and
it worked as expected so this may not be file system related.

Is there a way you can try another filesystem and mount it as root / then proceed with the upgrade?
I just want to rule out btrfs..

(In reply to Leo Sandoval from comment #16)
> (In reply to Mikhail from comment #15)
> > Created attachment 2121958 [details]
> > video demonstration
> > 
> > Following your request, I installed grub2-2.12-50.fc44 from Koji with GRUB
> > debug enabled (debug=all,-lexer,-scripting, pager=1) and reproduced the
> > issue again.
> 
> how did you enable the logging? using grub2-editenv? In theory a LOT of msgs 
> should be printed.
> 
> > 
> > > $ lsblk -f # place it on a pastebin, never expire
> > 
> > root@primary-ws ~# lsblk -f
> > NAME        FSTYPE FSVER LABEL    UUID                                
> > FSAVAIL FSUSE% MOUNTPOINTS
> > zram0       swap   1     zram0    1868a0d0-ac55-401b-b589-f856034c9dae      
> > [SWAP]
> > nvme1n1                                                                     
> > 
> > ├─nvme1n1p1 vfat   FAT16          4307-D37D                            
> > 469.9M     4% /boot/efi
> > ├─nvme1n1p2 swap   1              49251da1-a5cf-49ab-af4e-6dff17173e89      
> > [SWAP]
> > └─nvme1n1p3 btrfs        btrfs.37 b4996eda-3a45-4f45-ad59-be0bac273396   
> > 194G    37% /
> > nvme0n1     btrfs                 46ed41f1-2512-4e13-a502-4add0e26a341   
> > 5.6T    80% /home
> > root@primary-ws ~# 
> > 
> > 
> > > $ sudo reboot   # in the same pastebin as above, place any boot error seeing
> > 
> > No error messages are shown before it drops to the grub> prompt. Please see
> > the attached video (output_vbr_6.mp4).
> 
> no debug msgs shown which means that either is failing quite soon or debug
> is not properly
> enable for some reason. 

Ignore my previous comment. The reason of the no-msgs on console is that grub.cfg 
controlled by a btrfs filesystem is obviously not working. Instead, I will share 
a set of rpms with grub instrumented to print msgs.

> 
> BTW, I tried the same partition setup (btrfs on / and no /boot partition)
> but on VM and
> it worked as expected so this may not be file system related.
> 
> Is there a way you can try another filesystem and mount it as root / then
> proceed with the upgrade?
> I just want to rule out btrfs..

Mikhail and team, 

can you please do the following

We have two regression [1,2] with the proposed (and merged) fix [3] so unfortunately
this is not yet over: During the test day [TestDay], most testers booted fine the iso
so we merged [3]. During the previous weekend (10-11 Jan 2026), people
found boot issues [1] so [3] was untagged [4]. We are suspecting that this is something
related to the filesystem (btrfs?) and this is why during the test day using
a Live OS did not catch it. Test Day was indeed useful, fix worked but not for all
systems so clearly the fix needs to be revisited.

I ask you the following: From a stable system/grub, can you (RE)INSTALL these instructions:
  
$ sudo dnf install -y koji
$ mkdir tmp & cd tmp
$ koji download-task 140906146 # aka [5], this is just a scratch build just for x86_64, disable SB
$ sudo dnf install *.rpm
$ sudo reboot
  
The fix [5] contains  [3] plus some instrumentation enabling debugging ('debug=all,-lexer,-scripting').
In case of FAILURE (you boot but ended up in the grub prompt instead of the grub menu) please
attach a video recording all the boot process and also in the comment indicate your block/partition/fs
scheme with 'lsblk -f'.

I am also preparing another test-day which will basically do the above steps but I will wait hoping
that we will get enough results here first.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2427945
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2422881
[3] https://src.fedoraproject.org/rpms/grub2/pull-request/198
[4] https://forge.fedoraproject.org/releng/tickets/issues/13157
[5] https://koji.fedoraproject.org/koji/taskinfo?taskID=141063848
[TestDay] https://fedoraproject.org/wiki/Test_Day:2025-12-15_GRUB_out_of_memory_verification

(In reply to Leo Sandoval from comment #14)
> Following Marta request but being a bit more general of the info we believe
> would be useful, please do the following 
> 
> Assuming you are on a running rawhide systems, enable grub debug and upgrade
> to 2.12-50 with the following
> 
> $ sudo dnf install -y koji
> $ grub2-editenv - set debug=all,-lexer,-scripting
> $ grub2-editenv - set pager=1
> $ mkdir grub2-2.12-50
> $ cd grub2-2.12-50
> $ koji download-build --arch=x86_64 --arch=noarch grub2-2.12-50.fc44
> $ sudo dnf install *.rpm
> $ lsblk -f # place it on a pastebin, never expire
> $ sudo reboot   # in the same pastebin as above, place any boot error seeing
> $ #optional, just in case your system boots fine and you want to back to
> stable 'sudo dnf downgrade grub2\*'

I didn't install 2.12-50 again yet. Is that supposed to write debug info to a log file somewhere or show it at boot? Because the only info that was ever shown only flashed by for a split second and scrolled quickly so I wouldn't be able to catch any of that. If it's supposed to write to disk then that won't work because it was only able to access any of the disks after one of the times I rebooted.

Here's the lsblk info though.

NAME        FSTYPE FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
zram0       swap   1     zram0    bafba1a7-e023-4f93-a2cb-75456ad91514                [SWAP]
nvme2n1                                                                               
└─nvme2n1p1 btrfs        Home     12b1c0c9-c464-4f5b-9284-92609ba9f461  367.5G    80% /home
nvme0n1                                                                               
├─nvme0n1p1 vfat   FAT16          13F7-F971                               182M     9% /boot/efi
├─nvme0n1p2 btrfs        fedora   786a165f-7ee8-43aa-b2b7-374ff51e17b6  122.6G    13% /
├─nvme0n1p3                                                                           
└─nvme0n1p4 ntfs         Windows  103E4F3C3E4F1A5C                                    
nvme1n1     btrfs        Projects e8d03978-eba5-4376-a064-018b8560868e    1.6T    55% /run/media/mark/Projects

(In reply to Mark Grgurev from comment #19)
> (In reply to Leo Sandoval from comment #14)
> > Following Marta request but being a bit more general of the info we believe
> > would be useful, please do the following 
> > 
> > Assuming you are on a running rawhide systems, enable grub debug and upgrade
> > to 2.12-50 with the following
> > 
> > $ sudo dnf install -y koji
> > $ grub2-editenv - set debug=all,-lexer,-scripting
> > $ grub2-editenv - set pager=1
> > $ mkdir grub2-2.12-50
> > $ cd grub2-2.12-50
> > $ koji download-build --arch=x86_64 --arch=noarch grub2-2.12-50.fc44
> > $ sudo dnf install *.rpm
> > $ lsblk -f # place it on a pastebin, never expire
> > $ sudo reboot   # in the same pastebin as above, place any boot error seeing
> > $ #optional, just in case your system boots fine and you want to back to
> > stable 'sudo dnf downgrade grub2\*'
> 
> I didn't install 2.12-50 again yet. Is that supposed to write debug info to
> a log file somewhere or show it at boot?

grub2-2.12-50.fc44 is the build that was untagged, it has the fix. In the above instructions
I indicated enabling grub's debug but this WILL NOT work as I indicated in comment 17 (https://bugzilla.redhat.com/show_bug.cgi?id=2427945#c17).


> Because the only info that was ever
> shown only flashed by for a split second and scrolled quickly so I wouldn't
> be able to catch any of that. If it's supposed to write to disk then that
> won't work because it was only able to access any of the disks after one of
> the times I rebooted.

Right, so please install the Fedora OS again and follow steps from comment 18 (https://bugzilla.redhat.com/show_bug.cgi?id=2427945#c18)

Sorry for the confusion.

> 
> Here's the lsblk info though.
> 
> NAME        FSTYPE FSVER LABEL    UUID                                
> FSAVAIL FSUSE% MOUNTPOINTS
> zram0       swap   1     zram0    bafba1a7-e023-4f93-a2cb-75456ad91514      
> [SWAP]
> nvme2n1                                                                     
> 
> └─nvme2n1p1 btrfs        Home     12b1c0c9-c464-4f5b-9284-92609ba9f461 
> 367.5G    80% /home
> nvme0n1                                                                     
> 
> ├─nvme0n1p1 vfat   FAT16          13F7-F971                              
> 182M     9% /boot/efi
> ├─nvme0n1p2 btrfs        fedora   786a165f-7ee8-43aa-b2b7-374ff51e17b6 
> 122.6G    13% /
> ├─nvme0n1p3                                                                 
> 
> └─nvme0n1p4 ntfs         Windows  103E4F3C3E4F1A5C                          
> 
> nvme1n1     btrfs        Projects e8d03978-eba5-4376-a064-018b8560868e   
> 1.6T    55% /run/media/mark/Projects

Created attachment 2122093 [details]
video demonstration

(In reply to Leo Sandoval from comment #18)
> Mikhail and team, 
> 
> can you please do the following
> 
> We have two regression [1,2] with the proposed (and merged) fix [3] so
> unfortunately
> this is not yet over: During the test day [TestDay], most testers booted
> fine the iso
> so we merged [3]. During the previous weekend (10-11 Jan 2026), people
> found boot issues [1] so [3] was untagged [4]. We are suspecting that this
> is something
> related to the filesystem (btrfs?) and this is why during the test day using
> a Live OS did not catch it. Test Day was indeed useful, fix worked but not
> for all
> systems so clearly the fix needs to be revisited.
> 
> I ask you the following: From a stable system/grub, can you (RE)INSTALL
> these instructions:
>   
> $ sudo dnf install -y koji
> $ mkdir tmp & cd tmp
> $ koji download-task 140906146 # aka [5], this is just a scratch build just

Small correction: the Koji task id for [5] is 141063848 (not 140906146).
I used: koji download-task 141063848

> for x86_64, disable SB
> $ sudo dnf install *.rpm
> $ sudo reboot
>   
> The fix [5] contains  [3] plus some instrumentation enabling debugging
> ('debug=all,-lexer,-scripting').
> In case of FAILURE (you boot but ended up in the grub prompt instead of the
> grub menu) please
> attach a video recording all the boot process and also in the comment
> indicate your block/partition/fs
> scheme with 'lsblk -f'.


Unfortunately the scratch build [5] still drops to the `grub>` prompt (no menu).
Also, printing debug to the screen is not very actionable in this case:
the output is extremely verbose and scrolls too fast, so parts of it are unreadable in real time.
I recorded the whole boot attempt; please see the attached video: output_vbr_8.mp4.
Alternatively, if there is a supported way to capture GRUB debug output externally (console capture),
I can try that — screen output is too fast to read.

(In reply to Mikhail from comment #21)
> Created attachment 2122093 [details]
> video demonstration
> 
> (In reply to Leo Sandoval from comment #18)
> > Mikhail and team, 
> > 
> > can you please do the following
> > 
> > We have two regression [1,2] with the proposed (and merged) fix [3] so
> > unfortunately
> > this is not yet over: During the test day [TestDay], most testers booted
> > fine the iso
> > so we merged [3]. During the previous weekend (10-11 Jan 2026), people
> > found boot issues [1] so [3] was untagged [4]. We are suspecting that this
> > is something
> > related to the filesystem (btrfs?) and this is why during the test day using
> > a Live OS did not catch it. Test Day was indeed useful, fix worked but not
> > for all
> > systems so clearly the fix needs to be revisited.
> > 
> > I ask you the following: From a stable system/grub, can you (RE)INSTALL
> > these instructions:
> >   
> > $ sudo dnf install -y koji
> > $ mkdir tmp & cd tmp
> > $ koji download-task 140906146 # aka [5], this is just a scratch build just
> 
> Small correction: the Koji task id for [5] is 141063848 (not 140906146).
> I used: koji download-task 141063848

Right, thanks.


> 
> > for x86_64, disable SB
> > $ sudo dnf install *.rpm
> > $ sudo reboot
> >   
> > The fix [5] contains  [3] plus some instrumentation enabling debugging
> > ('debug=all,-lexer,-scripting').
> > In case of FAILURE (you boot but ended up in the grub prompt instead of the
> > grub menu) please
> > attach a video recording all the boot process and also in the comment
> > indicate your block/partition/fs
> > scheme with 'lsblk -f'.
> 
> 
> Unfortunately the scratch build [5] still drops to the `grub>` prompt (no
> menu).

yes, expected, I just enabled debugging through code.

> Also, printing debug to the screen is not very actionable in this case:
> the output is extremely verbose and scrolls too fast, so parts of it are
> unreadable in real time.
> I recorded the whole boot attempt; please see the attached video:
> output_vbr_8.mp4.
> Alternatively, if there is a supported way to capture GRUB debug output
> externally (console capture),
> I can try that — screen output is too fast to read.

unfortunately there is no other way than recording, e.g. no way to dump it into file.

From the video, the partition table (GPT) is not found (no partitions are found) thus grub
cannot read any file e.g grub.cfg, thus the fallback is to go into the grub prompt.

I'll let you know the next debugging step ASAP.

@Mark Grgurev
can your provide the motherboard/brand-model you are using?

So far we got these affected systems

* HW with OOM issues

- Asus ProArt Z790-CREATOR WIFI motherboard
- HP ZBook Fury 16 G9 Mobile Workstation PC/89C6
- Asus ROG MAXIMUS Z690 HERO
- ASUSTeK COMPUTER INC ROG MAXIMUS Z790 HERO

* HW with FS issues since 'OOM fix' (grub2-2.12-50), this ticket

- ASUS ROG STRIX B650E-I GAMING WIFI
- ASROCK B650I Lightning WiFi

With grub2-common-1:2.12-52.fc44.noarch we still see this almost same error in the image refreshes in anaconda-webui CI.

    Jan 26 12:26:46 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Modules.Storage[6490]: INFO:program:Running in chroot '/mnt/sysroot'... gen_grub_cfgstub /boot/grub2 /boot/efi/EFI/fedora
    org.fedoraproject.Anaconda.Modules.Storage[6490]: INFO:program:grub2-probe: error: ../grub-core/kern/fs.c:grub_fs_probe:123:unknown filesystem.
    Jan 26 12:26:46 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Modules.Storage[6490]: DEBUG:program:Return code of gen_grub_cfgstub: 1

I am attaching also the journal log of the failing installation.

Created attachment 2123806 [details]
journal

(In reply to Katerina Koukiou from comment #24)
> With grub2-common-1:2.12-52.fc44.noarch we still see this almost same error
> in the image refreshes in anaconda-webui CI.
> 
>     Jan 26 12:26:46 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com
> org.fedoraproject.Anaconda.Modules.Storage[6490]: INFO:program:Running in
> chroot '/mnt/sysroot'... gen_grub_cfgstub /boot/grub2 /boot/efi/EFI/fedora
>     org.fedoraproject.Anaconda.Modules.Storage[6490]:
> INFO:program:grub2-probe: error:
> ../grub-core/kern/fs.c:grub_fs_probe:123:unknown filesystem.
>     Jan 26 12:26:46 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com
> org.fedoraproject.Anaconda.Modules.Storage[6490]: DEBUG:program:Return code
> of gen_grub_cfgstub: 1
> 
> I am attaching also the journal log of the failing installation.

Did you mean to comment on https://bugzilla.redhat.com/show_bug.cgi?id=2429501 instead ? This issue is related to a regression introduced by a fix to fix an OOM issue.

Hi team,

we have a new fix [1] and hopefully this time would be the final one. A bit of resume
of this peculiar issue: the OOM issue is seen on those system with limited memory pool
so we try to increase the memory pool for grub_malloc [2] but hit other (unexpected) issues [3] so
[2] was reverted.

Before going into a second test-day for this OOM issue (if needed), I have proposed two testing scenarios below.
For both, please use a tesing HW (NEVER on your working desktop/lap) and DISABLE Secure boot (unsigned GRUB binaries
for the moment) temporaly.

- Option 1: ISO install (for those that cannot install Fedora due to OOM issues)
  0. Disable Secure boot
  1. Download the ISO [5] which contains the fix, flash it on a USB stick
  3. boot with it your HW. In theory the OOM would be seen here on those particular machines where OOM has been observed before
  4. If possible and your time allows, please install Fedora (use 'On the Network' in the Installation Source menu, this is just
     in case the auto-detect installation media fails). The reason we are asking for full installation is that last time we did not
     testers this step and we ended up with [3].
  5. Reboot your new Fedora System
  6. Follow instructions below (Option 2).  
  7. Share your results.

- Option 2: RPM install (for those already running a Fedora rawhide system)
  0. Disable Secure boot
  1. Boot your system
  2. Download the rpms[4], e.g. koji download-task --arch=x86_64 --arch=noarch 141497105
  3. install the rpms, e.g. sudo dnf install *.rpm
  4. Reboot
  5. Share results

[1] https://src.fedoraproject.org/rpms/grub2/pull-request/207
[2] https://src.fedoraproject.org/rpms/grub2/pull-request/198
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2427945
[4] https://koji.fedoraproject.org/koji/taskinfo?taskID=141497105
[5] https://people.redhat.com/lsandova/oom/boot-efi-alloc-on-verifiers.iso