Bug 1699761 - 'error: ../../grub-core/fs/fat.c:447:not a FAT filesystem' displayed briefly during boot
Summary: 'error: ../../grub-core/fs/fat.c:447:not a FAT filesystem' displayed briefly ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: 30
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1678718 1688643 1707084 1709378 1711293 1723057 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-15 08:00 UTC by Milan Zink
Modified: 2020-02-21 22:24 UTC (History)
33 users (show)

Fixed In Version: grub2-2.02-80.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-17 01:04:49 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Milan Zink 2019-04-15 08:00:46 UTC
Description of problem:
'error: ../../grub-core/fs/fat.c:447:not a FAT filesystem' displayed briefly during boot. Lenovo t480
UEFI, secure boot disabled

Version-Release number of selected component (if applicable):
[install@localhost ~]$ cat /etc/fedora-release 
Fedora release 30 (Thirty)
[install@localhost ~]$ rpm -qa | grep grub
grub2-efi-x64-cdboot-2.02-75.fc30.x86_64
grub2-efi-x64-2.02-75.fc30.x86_64
grub2-pc-modules-2.02-75.fc30.noarch
grub2-tools-minimal-2.02-75.fc30.x86_64
grub2-efi-ia32-2.02-75.fc30.x86_64
grub2-common-2.02-75.fc30.noarch
grub2-tools-extra-2.02-75.fc30.x86_64
grub2-tools-2.02-75.fc30.x86_64
grubby-8.40-30.fc30.x86_64
grub2-tools-efi-2.02-75.fc30.x86_64
grub2-pc-2.02-75.fc30.x86_64
grub2-efi-ia32-cdboot-2.02-75.fc30.x86_64

How reproducible:
Watch boot process

Steps to Reproduce:
Install F30 on Lenovo t480

Actual results:
'error: ../../grub-core/fs/fat.c:447:not a FAT filesystem' displayed briefly

Expected results:
No errors

Additional info:
Same problem on different HW - https://forums.fedoraforum.org/showthread.php?321108-F30-Boots-With-Message

Comment 1 dario 2019-04-16 09:33:24 UTC
Same error on Silverblue 30.20190413.n.1

Comment 2 Bartosz Nitkiewicz 2019-04-18 08:53:30 UTC
I have this error both on FB 30 and SB 30. I managed to fix it by removing all grub2 packages and use bootctl --path=/boot install. But it is just temporary solution.

Comment 3 Damián Barberón 2019-04-20 03:59:08 UTC
I have the same problem after clean install F30 Beta. 

I suspect it may be an F30 anaconda installer issue, b/c the grub error does not appear when upgrading F29 -> F30.

Comment 4 Milan Zink 2019-04-23 07:35:31 UTC
Possibly, problem is not present on machine upgraded from F29 -> F30.

Comment 5 Federico Chiacchiaretta 2019-04-23 10:28:47 UTC
Just upgraded from F29 -> F30, HP Probook 440 G3, UEFI with secureboot enabled, error message is displayed briefly on every boot.

Best,
Federico Chiacchiaretta

Comment 6 Damián Barberón 2019-04-24 01:48:03 UTC
Installed again F30 workstation from daily snapshot -20190422.n.1-, and I cannot reproduce the error message at boot.

Comment 7 Bartosz Nitkiewicz 2019-04-24 16:31:36 UTC
Found solution at least for pure Fedora 30 (don't know how to do it on Silverblue).

I've comment out GRUB_ENABLE_BLSCFG=true line in /etc/default/grub and then regenerate config:

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

No more errors on boot, but I don't know what that function does. In my case everything works normal.

Comment 8 Milan Zink 2019-04-25 06:29:31 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1599445
https://www.spinics.net/lists/fedora-devel/msg252433.html

Seems that new 'bootloaderspec' has still some issues.

Comment 9 Alexander Doelz 2019-05-01 11:28:49 UTC
I have the same bug after upgrading to Fedora 30 release on a Fujitsu A357 with UEFI-Boot.

Comment 10 Chris Tao 2019-05-01 15:17:36 UTC
Same issue after upgrading from f29. It's UEFI with BLS enabled.

Comment 11 Chris Tao 2019-05-02 05:16:33 UTC
Seems like unhiding the grub menu makes it goes directly into the boot selection screen. No error messages are displayed

# grub2-editenv - unset menu_auto_hide

Comment 12 Kelvin J. Hill 2019-05-03 13:14:42 UTC
(In reply to Chris Tao from comment #10)
> Same issue after upgrading from f29. It's UEFI with BLS enabled.

Same here - ASUS Z-170A motherboard.

Comment 13 Pascal Mathis 2019-05-04 12:11:22 UTC
Had the same issue here after upgrading from F29 to F30 with UEFI and enabled secure boot. My first attempt was to regenerate the GRUB configuration ("grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg") which unfortunately did not work. However after manually running "grub2-switch-to-blscfg" as well, the issue disappeared on my system.

Comment 14 Alexander Doelz 2019-05-04 12:45:46 UTC
(In reply to Pascal Mathis from comment #13)
> My first attempt was to regenerate the GRUB
> configuration ("grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg") which
> unfortunately did not work. However after manually running
> "grub2-switch-to-blscfg" as well, the issue disappeared on my system.

This didn't work for me.

Comment 15 adlo 2019-05-08 03:05:02 UTC
Upgraded from Fedora 29 to 30 and I have this problem.

Comment 16 Pavlo Rudyi 2019-05-08 15:27:01 UTC
(In reply to Bartosz Nitkiewicz from comment #7)

> I've comment out GRUB_ENABLE_BLSCFG=true line in /etc/default/grub and then
> regenerate config:
> 
> grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
> 

Thanks, it works.

Comment 17 Chris Murphy 2019-05-10 01:00:35 UTC
Huh, I'm not seeing this on any of three different UEFI systems, or a UEFI VM (qemu-kvm). Two are clean installed, two are upgrades. All use BLS (the default). It would be great to understand this problem and come up with a proper fix rather than a work around, in particular disabling BLSCFG seems long term untenable.

Is anyone able to parse this and why the error message happens? What is the test for GRUB_ERR_BAD_FS or what volume is even being tested? That is, is it really complaining about the EFI system partition? Or is it confused and complaining that e.g. /boot isn't FAT?
http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/fat.c

Comment 18 Damián Barberón 2019-05-10 02:03:57 UTC
grub_fat_mount (grub_disk_t disk) is the function that, at some point in the code, causes the jump to "goto fail;". After that, the grub_error (GRUB_ERR_BAD_FS, "not a FAT filesystem") is displayed.


I hope that my analysis will contribute to this issue.

Comment 19 Damián Barberón 2019-05-10 02:07:53 UTC
There are 16 conditions in the code that causes the jump to "goto fail".

Comment 20 Chris Murphy 2019-05-10 03:30:51 UTC
(In reply to Damián Barberón from comment #19)
> There are 16 conditions in the code that causes the jump to "goto fail".

Hmm, the error message isn't verbose enough to know which of the 16 conditions it is. And also, it grub-core/fs/fat.c is not recently modified, so it seems like a really misleading error. So if it is blscfg.mod, how can it cause a bogus "not FAT" error?

The source for blscfg.mod
https://github.com/rhboot/grub2/blob/fedora-30/grub-core/commands/blscfg.c

But maybe someone who still has this problem is willing to try and track it down by modifying the grub.cfg

- Add 'set debug=all' right after 'set pager=1'
- Add 'set debug=off' right before 'function load_video {'  ## approximately line 48

That will cause quite a bit of verbose debug output on the next boot, and you'll have to step through multiple pages (I got about 10 pages in a VM), and look for the error message. There might be a clue in the pages before the error shows up; so you might have to step through to get to the menu, boot, and reboot again, and step through to find the page before the error (good idea to count pages as you go through so this step is easier).

In default Fedora installations, only UEFI installations use FAT, and the only files read from FAT (the EFI System volume) is grub.cfg and grubenv. The actual blscfg snippets are not on FAT, they're in /boot/loader/entries which on a default installation is ext4.

Comment 21 Federico Chiacchiaretta 2019-05-10 16:17:57 UTC
(In reply to Chris Murphy from comment #20)
> ...
> But maybe someone who still has this problem is willing to try and track it
> down by modifying the grub.cfg
> 
> - Add 'set debug=all' right after 'set pager=1'
> - Add 'set debug=off' right before 'function load_video {'  ## approximately
> line 48
> ...

I tried turning on debug as suggested, but error does not generate in those lines.
I included all the section between "### BEGIN /etc/grub.d/10_linux ###" and "### END /etc/grub.d/10_linux ###", and I can tell that error is generated when "blscfg" is called in grub2-efi.cfg (line 129 in my config), right after "insmod blscfg".

I'll try to summarize the output, I took pictures of every page if you need to look at all the output.

When module starts, it detects ext2 filesystem at hd0,gpt2 and loops opening all the files in /boot/loader/entries.
After reading each file, it says:

commands/blscfg.c:380: Add entry with id "uuid-kernelver.fc30.x84_64"

Then it creates menu entries for each of the entries added before:

commands/blscfg.c:1020: bls_create_entries Creating entries from bls
commands/blscfg.c:713: create_entry got here
...omissis...
commands/blscfg.c:833: Added entry # id:"uuid-kernelver.fc30.x84_64"

The last entry added is "uuid-0-rescue" in my case, and 'error: ../../grub-core/fs/fat.c:447:not a FAT filesystem' is printed just after the last "Added entry..." line, then output follows with reading next line in grub2-efi.cfg:

script/lexer.c:321: token 281 text [if]
script/lexer.c:321: token 289 text [[]
script/lexer.c:321: token 289 text [-s]
script/lexer.c:321: token 289 text []
script/lexer.c:321: token 289 text [prefix]
script/lexer.c:321: token 289 text [/grubenv]
.... etc.

If you need more info, output or other tests, just let me know.

Best,
Federico Chiacchiaretta

Comment 22 Chris Murphy 2019-05-10 16:43:58 UTC
Interesting, so it happens rather late in grub.cfg execution. 

I see grub2-2.02-79.fc30 has gone stable. This post suggests that fixes it. Can people having the problem confirm/deny?
https://ask.fedoraproject.org/t/error-grub-core-fs-fat-cnot-a-fat-filesystem/785/6

Comment 23 Kelvin J. Hill 2019-05-10 16:50:43 UTC
(In reply to Chris Murphy from comment #22)
> Interesting, so it happens rather late in grub.cfg execution. 
> 
> I see grub2-2.02-79.fc30 has gone stable. This post suggests that fixes it.
> Can people having the problem confirm/deny?
> https://ask.fedoraproject.org/t/error-grub-core-fs-fat-cnot-a-fat-filesystem/
> 785/6

It does not.

Comment 24 Kelvin J. Hill 2019-05-10 17:03:52 UTC
(In reply to Kelvin J. Hill from comment #23)
> (In reply to Chris Murphy from comment #22)
> > Interesting, so it happens rather late in grub.cfg execution. 
> > 
> > I see grub2-2.02-79.fc30 has gone stable. This post suggests that fixes it.
> > Can people having the problem confirm/deny?
> > https://ask.fedoraproject.org/t/error-grub-core-fs-fat-cnot-a-fat-filesystem/
> > 785/6
> 
> It does not.

Perhaps I should add:

This is a dual boot system with the following configuration. 
Drive 0 (/dev/sda) Fedora 30 updated in stages,using the official method, from a clean installation of Fedora 27 under UEFI secure boot.
Drive 1 (/dev/sdb) Windows 10 Pro using legacy boot, not UEFI secure boot.

Comment 25 Javier Martinez Canillas 2019-05-10 18:00:11 UTC
I was able to reproduce this problem on a VM. I only see this when I don't have a boot partition (/boot is a directory in the root partition).

Is this the setup used by people that are facing this issue?

Comment 26 Kelvin J. Hill 2019-05-10 18:29:31 UTC
(In reply to Javier Martinez Canillas from comment #25)
> I was able to reproduce this problem on a VM. I only see this when I don't
> have a boot partition (/boot is a directory in the root partition).
> 
> Is this the setup used by people that are facing this issue?

No. I have both separate boot and efi partitions.

I tried a grub2-install to see if the new grub version would fix up anything bad. 

Big Mistake... It rendered my system unbootable with secure boot violation. Restoring from a backup bit-level drive image as I speak. I will NOT be doing that again.

Comment 27 Javier Martinez Canillas 2019-05-10 18:38:39 UTC
(In reply to Kelvin J. Hill from comment #26)
> (In reply to Javier Martinez Canillas from comment #25)
> > I was able to reproduce this problem on a VM. I only see this when I don't
> > have a boot partition (/boot is a directory in the root partition).
> > 
> > Is this the setup used by people that are facing this issue?
> 
> No. I have both separate boot and efi partitions.
> 

Thanks for the information.

> I tried a grub2-install to see if the new grub version would fix up anything
> bad. 
> 
> Big Mistake... It rendered my system unbootable with secure boot violation.
> Restoring from a backup bit-level drive image as I speak. I will NOT be
> doing that again.

Yes, I don't think that is recommended to run grub2-install for EFI installs.

Comment 28 Chris Murphy 2019-05-10 18:43:28 UTC
(In reply to Kelvin J. Hill from comment #26)
> Big Mistake... It rendered my system unbootable with secure boot violation.
> Restoring from a backup bit-level drive image as I speak. I will NOT be
> doing that again.

'grub2-install' should not be used with UEFI firmware. To reinstall GRUB, see the UEFI section to reinstall grub2 and shim packages.

https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader/#sec-Reinstalling_GRUB_2

Comment 29 Federico Chiacchiaretta 2019-05-10 19:49:34 UTC
(In reply to Chris Murphy from comment #22)

> I see grub2-2.02-79.fc30 has gone stable. This post suggests that fixes it.
> Can people having the problem confirm/deny?
> https://ask.fedoraproject.org/t/error-grub-core-fs-fat-cnot-a-fat-filesystem/
> 785/6

Issue is not fixed after update.

(In reply to Javier Martinez Canillas from comment #25)
> I was able to reproduce this problem on a VM. I only see this when I don't
> have a boot partition (/boot is a directory in the root partition).
> 
> Is this the setup used by people that are facing this issue?

Separate partition for boot (/dev/sda2) and efi (/dev/sda1) here too.
I should add that I have a separate disk (/dev/sdb) for a Windows 10 install, but I have "GRUB_DISABLE_OS_PROBER=true" in /etc/sysconfig/grub, so there is no entry for Windows 10 in grub menu.

Comment 30 Michael J Gruber 2019-05-14 07:09:56 UTC
I see this problem on F30 upgraded from F29 after enabling menu_auto_hide (for the new boot process hotness); probably it was there before (but hidden by the menu). The filesystems are:

/boot     ext4
/boot/efi vfat
/         btrfs

secure boot is working fine nonetheless (including the seemless throbber hotness). It seems as if grub is expecting a vfat boot?

Note that I started from a fresh grubenv (grub2-editenv), used grub2-mkconfig to recreate the config (/etc/grub2-efi.cfg links to /boot/efi/EFI/fedora/grub.cfg as it should) and recerated the initrd after changing the plymouth theme.

Comment 31 Michael J Gruber 2019-05-14 11:17:54 UTC
Throwing in another data point: My /etc/grub2-efi.cfg gets generated with an entry

### BEGIN /etc/grub.d/10_linux ###
insmod part_gpt
insmod ext2
set root='hd0,gpt7'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt7 --hint-efi=hd0,gpt7 --hint-baremetal=ahci0,gpt7  4413821b-622d-458c-b839-9f963ad2cef3
else
  search --no-floppy --fs-uuid --set=root 4413821b-622d-458c-b839-9f963ad2cef3
fi
insmod part_gpt
insmod fat
set boot='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=boot --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  C28D-513C
else
  search --no-floppy --fs-uuid --set=boot C28D-513C
fi

etc., that is the Fedora entry (/dev/sda7 is /boot ext4) plus a remnant of the OEM Windows (dev/sda2 is /boot/efi vfat).

Then comes the blscfg part, and then (again):

### BEGIN /etc/grub.d/30_os-prober ###
menuentry 'Windows Boot Manager (on /dev/sda2)' --class windows --class os $menuentry_id_option 'osprober-efi-C28D-513C' {
        insmod part_gpt
        insmod fat
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  C28D-513C
        else
          search --no-floppy --fs-uuid --set=root C28D-513C
        fi
        chainloader /EFI/Microsoft/Boot/bootmgfw.efi
}

# Other OS found, undo autohiding of menu unless menu_auto_hide=2
if [ "${orig_timeout_style}" -a "${menu_auto_hide}" != "2" ]; then
  set timeout_style=${orig_timeout_style}
  set timeout=${orig_timeout}
fi
### END /etc/grub.d/30_os-prober ###

I kind of suspect that the first one is bogus, but maybe it (gpt2) should appear twice?

Comment 32 Chris Murphy 2019-05-14 17:39:09 UTC
(In reply to Michael J Gruber from comment #31)
> I kind of suspect that the first one is bogus, but maybe it (gpt2) should
> appear twice?

That's normal. I have the same thing on a virtually identical setup (HFS+ for the ESP on the macbook test system; ext4 boot; btrfs root).

/loader/entries is a valid locations for bls snippets on both the boot volume as well as the EFI system volume.

But consider this:

insmod blscfg
blscfg
if [ -s $prefix/grubenv ]; then
  load_env
fi

Since 10_linux sets up search on both the FAT ESP and ext4 boot, blscfg finds snippets in either location. But wouldn't load_env also search both locations for grubenv? Thing is, it shouldn't complain about loading grubenv no matter the file system. The only gotcha with grubenv is limited write support, for sure FAT and for now ext4. But XFS, Btrfs, LVM and RAID are off limits.

Why is this load_env  present in this location anyway? There's a much earlier load_env in 00_header.

Michael, can you edit your grub.cfg, 10_linux section near the end, by framing the blscfg command like this:

insmod blscfg
set debug=all
blscfg
set debug=off
if [ -s $prefix/grubenv ]; then
  load_env
fi

You will likely get pages of debug info before you see the grub menu, but you're looking for this "not a FAT filesystem" message buried within debug info. That would at least point to whether it's in blscfg or somewhere else. If you only see it after, you could try

insmod blscfg
blscfg
set debug=all
if [ -s $prefix/grubenv ]; then
  load_env
fi
set debug=off

And then that will only debug the loading of grubenv. Like I mention above, if you set debug=all for too big of a range, it ends up being super tedious to page through all the debug message.

My next suspect will be the 10_reset_boot_success section, at the very end it does save_env. And then 12_menu_auto_hide also does save_env.

Comment 33 Javier Martinez Canillas 2019-05-14 18:21:39 UTC
(In reply to Chris Murphy from comment #32)
> (In reply to Michael J Gruber from comment #31)
> > I kind of suspect that the first one is bogus, but maybe it (gpt2) should
> > appear twice?
> 
> That's normal. I have the same thing on a virtually identical setup (HFS+
> for the ESP on the macbook test system; ext4 boot; btrfs root).
> 
> /loader/entries is a valid locations for bls snippets on both the boot
> volume as well as the EFI system volume.
> 
> But consider this:
> 
> insmod blscfg
> blscfg
> if [ -s $prefix/grubenv ]; then
>   load_env
> fi
> 
> Since 10_linux sets up search on both the FAT ESP and ext4 boot, blscfg
> finds snippets in either location. But wouldn't load_env also search both
> locations for grubenv? Thing is, it shouldn't complain about loading grubenv
> no matter the file system. The only gotcha with grubenv is limited write
> support, for sure FAT and for now ext4. But XFS, Btrfs, LVM and RAID are off
> limits.
> 
> Why is this load_env  present in this location anyway? There's a much

Yes, that's bogus indeed. I don't know why there's a load_env there and we should remove it.

I had some time to dig on this bug. The blscfg module uses the grub_fs_probe() function to query the file system of the partition containing the BLS entries. And what this function does is to iterate over all supported files ystems and attempt to list the files in the root of the partition by using the fs struct grub_fs .dir callback. The callback that sucedes is the one for the file system used in that partition.

Now, each .dir callbacks first try to mount the file system, but the grub_fat_mount() function is particularly noisy and prints an error if a partition is can't be mounted with that file system. No other mount function does this.

We can see that if we try to execute the blscfg command from the grub prompt with debug=fs:


grub> set debug=fs
grub> blscfg
kern/fs.c:56: Detecting xfs...
kern/fs.c:78: xfs detection failed.
kern/fs.c:56: Detecting iso9660...
kern/fs.c:78: iso9660 detection failed.
kern/fs.c:56: Detecting hfsplus...
kern/fs.c:78: hfsplus detection failed.
kern/fs.c:56: Detecting fat...
kern/fs.c:78: fat detection failed.
kern/fs.c:56: Detecting ext2...
kern/fs.c:56: Detecting xfs...
kern/fs.c:78: xfs detection failed.
kern/fs.c:56: Detecting iso9660...
kern/fs.c:78: iso9660 detection failed.
kern/fs.c:56: Detecting hfsplus...
kern/fs.c:78: hfsplus detection failed.
kern/fs.c:56: Detecting fat...
kern/fs.c:78: fat detection failed.
kern/fs.c:56: Detecting ext2...
kern/fs.c:56: Detecting xfs...
kern/fs.c:78: xfs detection failed.
kern/fs.c:56: Detecting iso9660...
kern/fs.c:78: iso9660 detection failed.
grub>   ./../grub-core/fs/fat.c:447:not a FAT filesystem.

So I think that the solution is just to not make the fat implementation to print an error if a partition is tried to be mounted and fails. Since that's what GRUB uses to determine the file system used on a partition.

Comment 34 Javier Martinez Canillas 2019-05-14 21:11:28 UTC
*** Bug 1709378 has been marked as a duplicate of this bug. ***

Comment 35 Javier Martinez Canillas 2019-05-15 09:53:52 UTC
*** Bug 1678718 has been marked as a duplicate of this bug. ***

Comment 36 Javier Martinez Canillas 2019-05-15 09:54:54 UTC
*** Bug 1707084 has been marked as a duplicate of this bug. ***

Comment 37 Javier Martinez Canillas 2019-05-15 09:56:49 UTC
*** Bug 1688643 has been marked as a duplicate of this bug. ***

Comment 38 Javier Martinez Canillas 2019-05-15 10:15:01 UTC
(In reply to Javier Martinez Canillas from comment #33)
> (In reply to Chris Murphy from comment #32)
> > (In reply to Michael J Gruber from comment #31)
> > > I kind of suspect that the first one is bogus, but maybe it (gpt2) should
> > > appear twice?
> > 
> > That's normal. I have the same thing on a virtually identical setup (HFS+
> > for the ESP on the macbook test system; ext4 boot; btrfs root).
> > 
> > /loader/entries is a valid locations for bls snippets on both the boot
> > volume as well as the EFI system volume.
> > 
> > But consider this:
> > 
> > insmod blscfg
> > blscfg
> > if [ -s $prefix/grubenv ]; then
> >   load_env
> > fi
> > 
> > Since 10_linux sets up search on both the FAT ESP and ext4 boot, blscfg
> > finds snippets in either location. But wouldn't load_env also search both
> > locations for grubenv? Thing is, it shouldn't complain about loading grubenv
> > no matter the file system. The only gotcha with grubenv is limited write
> > support, for sure FAT and for now ext4. But XFS, Btrfs, LVM and RAID are off
> > limits.
> > 
> > Why is this load_env  present in this location anyway? There's a much
> 
> Yes, that's bogus indeed. I don't know why there's a load_env there and we
> should remove it.
>

I've removed this in grub2-2.02-80.fc30.
 
[snip]

> 
> So I think that the solution is just to not make the fat implementation to
> print an error if a partition is tried to be mounted and fails. Since that's
> what GRUB uses to determine the file system used on a partition.


I was wrong on this one. There are other FS mount functions that print an error but it's just that only the last error message set with grub_error() was printed.

The bug was in the blscfg module, it wrongly left the grub_errno variable set to an error code even when it succeeded and so the command dispatcher thought that it failed and printed the last error message that was set. I've fixed this in grub2-2.02-80.fc30.

Comment 39 Fedora Update System 2019-05-15 10:16:50 UTC
grub2-2.02-80.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-944ce0bb51

Comment 40 Milan Zink 2019-05-15 11:33:09 UTC
grub2-2.02-80.fc30  is fixing issue for me.

Comment 41 Chris Murphy 2019-05-15 21:54:39 UTC
(In reply to Javier Martinez Canillas from comment #33)

> Now, each .dir callbacks first try to mount the file system, but the
> grub_fat_mount() function is particularly noisy and prints an error if a
> partition is can't be mounted with that file system. No other mount function
> does this.

Strange that I don't see this in my identical setups, including FAT ESP, ext4 boot, Btrfs root - which should trigger this.

> So I think that the solution is just to not make the fat implementation to
> print an error if a partition is tried to be mounted and fails.

Yeah or move that error to a debug function so it only appears if debug is set. It could come in handy for something I guess, although it'd be a lot more helpful had that error reported a GRUB (device,partition).

Comment 42 Chris Murphy 2019-05-15 21:56:11 UTC
Ahaha, I just read c38, disregard c41. Pays to read the whole thread before replying, Murphy!

Comment 43 Fedora Update System 2019-05-16 02:36:26 UTC
grub2-2.02-80.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-944ce0bb51

Comment 44 Frank Ansari 2019-05-16 19:04:40 UTC
"Found solution at least for pure Fedora 30 (don't know how to do it on Silverblue).

I've comment out GRUB_ENABLE_BLSCFG=true line in /etc/default/grub and then regenerate config:

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

No more errors on boot, but I don't know what that function does. In my case everything works normal."

I tried this on Silverblue 30 and my PC refused to boot.

Fortunately I saved a copy of my original grub.cfg which I could reactivate with a recsue stick.

Comment 45 Chris Murphy 2019-05-16 21:23:06 UTC
(In reply to Frank Ansari from comment #44)
> I've comment out GRUB_ENABLE_BLSCFG=true line in /etc/default/grub and then
> regenerate config:
> 
> grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
> 
> No more errors on boot, but I don't know what that function does. In my case
> everything works normal."

It reverts to old bootloader behavior by disabling this Fedora 30 feature:
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault

Strictly speaking the system is now in a custom configuration. It should not be recommended as a solution to others, who likewise will have no idea what they've done, and that they are now using a non-standard configuration.

Comment 46 Fedora Update System 2019-05-17 01:04:49 UTC
grub2-2.02-80.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 47 Javier Martinez Canillas 2019-05-17 07:00:35 UTC
Hello Frank,

(In reply to Frank Ansari from comment #44)
> "Found solution at least for pure Fedora 30 (don't know how to do it on
> Silverblue).
> 
> I've comment out GRUB_ENABLE_BLSCFG=true line in /etc/default/grub and then
> regenerate config:
> 
> grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
> 
> No more errors on boot, but I don't know what that function does. In my case
> everything works normal."
>

As Chris said, this disables the BLS configuration. Instead please give it a try to the latest grub2 package that should fix the issue.

If you really want to disable the BLS configuration, then you need to install the grubby-deprecated package or otherwise no new boot entries will be added when kernel packages are installed.
 
> I tried this on Silverblue 30 and my PC refused to boot.
> 
> Fortunately I saved a copy of my original grub.cfg which I could reactivate
> with a recsue stick.

Comment 48 romain.cousture 2019-05-17 15:33:51 UTC
I confirm that after update of grub2-tools ; error disappeared

Comment 49 Frank Ansari 2019-05-17 19:31:33 UTC
OK - I did the update. Now these are all my grub packages:

[root@bat ~]# rpm -qa | grep grub
grub2-tools-2.02-80.fc30.x86_64
ostree-grub2-2019.2-1.fc30.x86_64
grub2-tools-extra-2.02-80.fc30.x86_64
grub2-common-2.02-80.fc30.noarch
grub2-pc-2.02-80.fc30.x86_64
grub2-tools-minimal-2.02-80.fc30.x86_64
grubby-8.40-30.fc30.x86_64
grub2-efi-x64-2.02-80.fc30.x86_64
grub2-pc-modules-2.02-80.fc30.noarch

After updating and rebooting I saw the same error as before.

Comment 50 Frank Ansari 2019-05-17 19:47:02 UTC
Either this is not fixed or my original bug report is not a duplicate bug to this.

I get the error described here:

https://bugzilla.redhat.com/show_bug.cgi?id=1707084

Comment 51 Javier Martinez Canillas 2019-05-17 23:32:04 UTC
(In reply to Frank Ansari from comment #50)
> Either this is not fixed or my original bug report is not a duplicate bug to
> this.
> 
> I get the error described here:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1707084

If you have a legacy BIOS install, you may need to do a grub2-install /dev/sda (replace sda for the block device where your grub is installed).

Comment 52 Frank Ansari 2019-05-18 16:43:33 UTC
Legacy? I am running on UEFI (I see a lot of stuff in /sys/firmware/efi/efivars).

Grub is working and I can boot my PC with it. It is just that most of the times I boot my PC I get the error described above.

This error was not there with Fedora 29 Silverblue. It was also not there after updating to Fedora 30 Silverblue.

It first appeared on a fresh installation of Fedora 30 Silverblue.

Comment 53 Frank Ansari 2019-05-18 16:55:47 UTC
"I have this error both on FB 30 and SB 30. I managed to fix it by removing all grub2 packages and use bootctl --path=/boot install. But it is just temporary solution."

From my perspective this should be the standard solution for UEFI systems (which are probably most PCs).

Is there a safe way to remove all this grub stuff and replace it with bootctl without ending up with a non-booting PC?

Before I used Silverblue I always removed all this grub stuff and used gummiboot but I am not sure whether this is recommended for Silverblue.

Comment 54 Chris Murphy 2019-05-18 18:48:27 UTC
(In reply to Frank Ansari from comment #52)
> Legacy? I am running on UEFI (I see a lot of stuff in
> /sys/firmware/efi/efivars).
> 
> Grub is working and I can boot my PC with it. It is just that most of the
> times I boot my PC I get the error described above.
> 
> This error was not there with Fedora 29 Silverblue. It was also not there
> after updating to Fedora 30 Silverblue.
> 
> It first appeared on a fresh installation of Fedora 30 Silverblue.

The problem with Silverblue is its bootloader is not updated automatically, even on UEFI. So if the original installation contains a bootloader with a bug, it's not fixable with a software update right now. You can manually copy the *.efi files from the usr directory to the EFI system volume - easiest to just do 'find /usr -name *.efi'

See bug 1293725

Comment 55 Frank Ansari 2019-05-18 21:03:39 UTC
I must admit my knowledge aobut rescuing a Silverblue system is very low until now. I tried to play with bootloader today and ended up in a new installation.

For example: how can I chroot a Silverblue system correctly? So far I have not a good idea about that.

Seems that Fedora sticks to grub instead of bootclt (gummiboot). Don't understand why.

Comment 56 Frank Ansari 2019-05-19 17:44:23 UTC
Anyway - I still have this issue:

When booting my system sometime (not always) there is this error:

setup.error: ../../grub-core/disk/loopback.c:165:can't open device.

When the error appears no boot menu is shown. When it does not appear I see the normal grub menu.

Comment 57 Chris Murphy 2019-05-20 05:24:48 UTC
(In reply to Frank Ansari from comment #56)
> Anyway - I still have this issue:
> 
> When booting my system sometime (not always) there is this error:
> 
> setup.error: ../../grub-core/disk/loopback.c:165:can't open device.
> 
> When the error appears no boot menu is shown. When it does not appear I see
> the normal grub menu.

Not the same bug, please file a new bug. Include

/boot/grub2/grub.cfg
/boot/grub2/grubenv
/etc/default/grub
/boot/loader/entries/*   ## all of them

And also output from

$ sudo efibootmgr -v     ## confirm whether UEFI and what bootloader it points to

Comment 58 Javier Martinez Canillas 2019-05-20 07:11:47 UTC
(In reply to Frank Ansari from comment #56)
> Anyway - I still have this issue:
> 
> When booting my system sometime (not always) there is this error:
> 
> setup.error: ../../grub-core/disk/loopback.c:165:can't open device.
> 
> When the error appears no boot menu is shown. When it does not appear I see
> the normal grub menu.

Just to be sure, this is with the latest version of grub? That is, you updated the grub EFI binary in the ESP of your Silverblue system as Chris mentioned?

Comment 59 Frank Ansari 2019-05-20 18:48:09 UTC
[(In reply to Javier Martinez Canillas from comment #58)
> (In reply to Frank Ansari from comment #56)
> > Anyway - I still have this issue:
> > 
> > When booting my system sometime (not always) there is this error:
> > 
> > setup.error: ../../grub-core/disk/loopback.c:165:can't open device.
> > 
> > When the error appears no boot menu is shown. When it does not appear I see
> > the normal grub menu.
> 
> Just to be sure, this is with the latest version of grub? That is, you
> updated the grub EFI binary in the ESP of your Silverblue system as Chris
> mentioned?

In Comment 49 you see my grub versions.

Here again:

[root@bat selinux]# rpm -qa | grep grub
grub2-pc-2.02-80.fc30.x86_64
grub2-pc-modules-2.02-80.fc30.noarch
grubby-8.40-30.fc30.x86_64
grub2-efi-x64-2.02-80.fc30.x86_64
grub2-tools-2.02-80.fc30.x86_64
ostree-grub2-2019.2-1.fc30.x86_64
grub2-tools-extra-2.02-80.fc30.x86_64
grub2-common-2.02-80.fc30.noarch
grub2-tools-minimal-2.02-80.fc30.x86_64

Comment 60 Chris Murphy 2019-05-20 19:15:08 UTC
(In reply to Frank Ansari from comment #59)
> 
> In Comment 49 you see my grub versions.
> 

But you don't say if you copied grubx64.efi found in /usr to replace the one in /boot/efi/EFI/fedora/. Updating Silverblue doesn't do this for you.

Comment 61 Frank Ansari 2019-05-20 19:45:08 UTC
(In reply to Chris Murphy from comment #60)
> (In reply to Frank Ansari from comment #59)
> > 
> > In Comment 49 you see my grub versions.
> > 
> 
> But you don't say if you copied grubx64.efi found in /usr to replace the one
> in /boot/efi/EFI/fedora/. Updating Silverblue doesn't do this for you.

Please find my answer here.

https://bugzilla.redhat.com/show_bug.cgi?id=1712137

Comment 62 Phil Coulson 2019-06-25 09:32:41 UTC
Hi all,

I'm still experiencing this issue with the most recent (as of yesterday) silverblue 30 iso

Comment 63 Javier Martinez Canillas 2019-06-25 09:42:34 UTC
(In reply to Phil Coulson from comment #62)
> Hi all,
> 
> I'm still experiencing this issue with the most recent (as of yesterday)
> silverblue 30 iso

That's because the EFI System Partition (ESP) isn't updated as a part of the ostree deployment transaction. I've proposed an 'ostree admin esp-upgrade' command but we are still discussing on the implementation:

https://github.com/ostreedev/ostree/pull/1873

For now you could copy everything that's in /usr/lib/ostree-boot/efi to /boot/efi in order to have the latest grub2.

If you have a legacy BIOS installation, then you need to execute 'grub2-install /dev/sda' (or whatever is the block device where grub was installed).

Comment 64 Phil Coulson 2019-06-25 11:09:59 UTC
thanks for the work around @Javier, that fixed it!

Comment 65 Javier Martinez Canillas 2019-10-15 06:58:03 UTC
*** Bug 1723057 has been marked as a duplicate of this bug. ***

Comment 66 Cody 2019-10-15 11:20:47 UTC
Yes, yes, that's great Javier.

The *FIRST REACTION* to a bug report reported almost *4 MONTHS AGO* - is to mark it as a duplicate. 

Hilariously though it's NOT a duplicate. It's not even remotely related. I figured out the problem and if you bothered reading my report you'd know this. Completely unrelated to GRUB. Yes there was an aspect to it but it wasn't what caused the rebooting. My bug report also pointed something else out that was rather .. ironic .. namely that the checksum file was a 404 error. Maybe that's been fixed by now but it was certainly unaddressed on my report (or additional comment).

Comment 67 Javier Martinez Canillas 2019-10-15 11:38:29 UTC
(In reply to Cody from comment #66)
> Yes, yes, that's great Javier.
> 
> The *FIRST REACTION* to a bug report reported almost *4 MONTHS AGO* - is to
> mark it as a duplicate. 
> 
> Hilariously though it's NOT a duplicate. It's not even remotely related. I
> figured out the problem and if you bothered reading my report you'd know
> this. Completely unrelated to GRUB. Yes there was an aspect to it but it
> wasn't what caused the rebooting. My bug report also pointed something else
> out that was rather .. ironic .. namely that the checksum file was a 404
> error. Maybe that's been fixed by now but it was certainly unaddressed on my
> report (or additional comment).

Yes I read your bug report but the only issue related to GRUB as far as I can tell that you reported was the "grub error in diskfilter.c for a brief moment before menu loads" issue. And that is a duplicate of this bug. I know hat was a red herring for you, but the bugzilla component was still set to GRUB and so I chose to close it as a duplicate for this bug.

Sorry that we didn't answer to your bug report before, but we have a lot of bug reports and try to do our best.

Feel free to re-open that Bug 1723057 and set the component to anything that you think is more suitable.

Comment 68 Cody 2019-10-15 11:46:09 UTC
(In reply to Javier Martinez Canillas from comment #67)

> 
> Yes I read your bug report but the only issue related to GRUB as far as I
> can tell that you reported was the "grub error in diskfilter.c for a brief
> moment before menu loads" issue. And that is a duplicate of this bug. I know
> hat was a red herring for you, but the bugzilla component was still set to
> GRUB and so I chose to close it as a duplicate for this bug.

Well maybe the error itself was, I don't know. The symptoms I'm not sure. But you're right that that was the component.

> 
> Sorry that we didn't answer to your bug report before, but we have a lot of
> bug reports and try to do our best.

No worries mate. I am very well aware of that I just am very very stressed right now and utterly exhausted. Those are the least of my worries right now.

> 
> Feel free to re-open that Bug 1723057 and set the component to anything that
> you think is more suitable.

Appreciate that comment. No. I'm afraid that it's not worth it for me. As it is I did a system upgrade anyway (though I haven't yet upgraded to F30 - I'll worry about this later, too much going on right now).

I apologise for the sarcasm. Typically I'm the one who encourages devs and users (for I am both) to work together rather than inadequate reporting and rowing. I am not remotely perfect though and I should not have said what I said. I think part of it is that some of the things in Fedora in recent years (in fact Red Hat) feel like regressions to me but some of this isn't fair and as I said I have a lot going on (four deaths in the family last year and those made worse by a number of other situations). I also was fighting with a raid6 install for many days with CentOS and that obviously doesn't help my mood.

Anyway thank you for the reply and I'm sorry for the scorn.

Comment 69 Javier Martinez Canillas 2020-01-28 14:00:42 UTC
*** Bug 1711293 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.