Bug 1729793 - Fedora 30 rendered unbootable on Libreboot devices due to GRUB2 changes
Summary: Fedora 30 rendered unbootable on Libreboot devices due to GRUB2 changes
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: 29
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-14 17:45 UTC by Demetris M
Modified: 2019-11-27 23:33 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-27 23:33:29 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
99-grub-mkconfig.install (1.56 KB, application/x-shellscript)
2019-10-15 10:27 UTC, Javier Martinez Canillas
no flags Details
99-grub-mkconfig.install (1.47 KB, application/x-shellscript)
2019-10-15 12:23 UTC, Javier Martinez Canillas
no flags Details

Description Demetris M 2019-07-14 17:45:28 UTC
Description of problem:
After upgrading a non-UEFI Fedora 29 system to Fedora 30 with the dnf plugin method, the Libreboot system (in this case a X200 laptop) reboots into an unbootable stage, as Libreboot cannot detect Fedora's grub.cfg and continue with booting the OS 

Version-Release number of selected component (if applicable):
N/A

How reproducible:
Only have one device to test, reproducible every time

Steps to Reproduce:
1. Upgrade a previously non-UEFI based Fedora system to Fedora 30
2. Reboot will occur once upgrades are applied


Actual results:
Libreboot's internal GRUB tries all its fallbacks end fails unable to detect a grub.cfg from which to boot

Expected results:
Libreboot's internal GRUB detects the grub.cfg in /boot/grub2 and loads its entries, as it did with Fedora 29

Additional info:
The "grub> configfile /grub2/grub.cfg.rpmsave" workaround in
https://fedoraproject.org/wiki/Common_F30_bugs#GRUB_boot_menu_is_not_populated_after_an_upgrade

allows me to boot the fc29 kernel, but the recommendation to run "grub2-install" again does not help.

Comment 1 Demetris M 2019-07-14 17:56:58 UTC
GRUB_ENABLE_BLSCFG=false and regeneration allows Libreboot to once again boot Fedora. 

Whether this is the appropriate solution is beyond my knowledge. Maybe the upgrade default should be to keep the old format if one is updating from non-UEFI set ups?

Comment 2 Javier Martinez Canillas 2019-07-15 08:22:34 UTC
(In reply to mdtha from comment #1)
> GRUB_ENABLE_BLSCFG=false and regeneration allows Libreboot to once again
> boot Fedora. 
> 
> Whether this is the appropriate solution is beyond my knowledge. Maybe the

Yes, this is the proper way to handle this in my opinion. Since you are not using Fedora's GRUB but instead Libreboot's GRUB which doesn't have BLS support.

Comment 3 Demetris M 2019-07-15 09:21:40 UTC
(In reply to Javier Martinez Canillas from comment #2)
> (In reply to mdtha from comment #1)
> > GRUB_ENABLE_BLSCFG=false and regeneration allows Libreboot to once again
> > boot Fedora. 
> > 
> > Whether this is the appropriate solution is beyond my knowledge. Maybe the
> 
> Yes, this is the proper way to handle this in my opinion. Since you are not
> using Fedora's GRUB but instead Libreboot's GRUB which doesn't have BLS
> support.

Thanks for looking into this. 

Do you have an opinion on how should the upgrade scripts handle this case? At the very least, in order to make sure that GRUB_ENABLE_BLSCFG will not be enabled in the next upgrade to Fedora 31 if it's set to false already, if not to protect 28 and 29 users from this surprise?

Comment 4 Javier Martinez Canillas 2019-07-15 09:36:44 UTC
(In reply to mdtha from comment #3)
> (In reply to Javier Martinez Canillas from comment #2)
> > (In reply to mdtha from comment #1)
> > > GRUB_ENABLE_BLSCFG=false and regeneration allows Libreboot to once again
> > > boot Fedora. 
> > > 
> > > Whether this is the appropriate solution is beyond my knowledge. Maybe the
> > 
> > Yes, this is the proper way to handle this in my opinion. Since you are not
> > using Fedora's GRUB but instead Libreboot's GRUB which doesn't have BLS
> > support.
> 
> Thanks for looking into this. 
> 
> Do you have an opinion on how should the upgrade scripts handle this case?
> At the very least, in order to make sure that GRUB_ENABLE_BLSCFG will not be
> enabled in the next upgrade to Fedora 31 if it's set to false already, if

That's already the case, if you have GRUB_ENABLE_BLSCFG=false set then the script that's run on upgrade to switch to a BLS configuration won't do anything. It'll only enable if you didn't opt-out.

> not to protect 28 and 29 users from this surprise?

That's more tricky. We can't support all the custom configurations that a user might have. Since you are not using the GRUB that we ship on Fedora, it might be possible that you experience this kind of incompatibilities.

By the way, if you use GRUB_ENABLE_BLSCFG=false then the grubby-deprecated package must be installed for new kernel entries to be added when a kernel package is installed.

Comment 5 Demetris M 2019-07-15 10:52:43 UTC
(In reply to Javier Martinez Canillas from comment #4)

> That's already the case, if you have GRUB_ENABLE_BLSCFG=false set then the
> script that's run on upgrade to switch to a BLS configuration won't do
> anything. It'll only enable if you didn't opt-out. [...]

Gotcha. It was a problem in 29->30 because that was a new change but it won't be when I do the next upgrade. 

Now, correct me if I am wrong, but that would be an unrecoverable bug for fresh installs, which don't have an old-style grub.cfg.rpmsave file to fall back to. My guess is that if one does a fresh install from a LiveUSB, they can chroot, install grubby-deprecated and switch BLS off, but if they do a net install or use the DVD image, they will be stuck. 

I can understand this configuration not being officially supported, but it should be mentioned in the Common Bugs wiki page at the very least.

Comment 6 Demetris M 2019-07-15 11:05:25 UTC
Libreboot bug URL: https://notabug.org/libreboot/libreboot/issues/657

Comment 7 Javier Martinez Canillas 2019-10-15 07:53:54 UTC
Is there a way to check from user-space if a machine is using Libreboot?

We already have some cases where BLS is not supported (i.e: ppc64le OPAL machines with an old firmware and Xen DomU guests), this is handled by the /usr/lib/kernel/install.d/99-grub-mkconfig.install script and in those cases a grub.cfg is re-generated with grub2-mkconfig.

Comment 8 Demetris M 2019-10-15 09:43:11 UTC
(In reply to Javier Martinez Canillas from comment #7)
> Is there a way to check from user-space if a machine is using Libreboot?


It looks possible, see the following 

----
> cat /sys/class/dmi/id/bios*
09/07/2016
coreboot
CBET4000 3774c98

----

Perhaps it will be harder to distinguish between versions of coreboot (of which libreboot is a downstream) with and without BLS support, if versions that support it exist.

Comment 9 Javier Martinez Canillas 2019-10-15 10:02:18 UTC
(In reply to mdtha from comment #8)
> (In reply to Javier Martinez Canillas from comment #7)
> > Is there a way to check from user-space if a machine is using Libreboot?
> 
> 
> It looks possible, see the following 
> 
> ----
> > cat /sys/class/dmi/id/bios*
> 09/07/2016
> coreboot
> CBET4000 3774c98
>

Interesting, coreboot will be set in the /sys/class/dmi/bios_vendor is that correct?
 
> ----
> 
> Perhaps it will be harder to distinguish between versions of coreboot (of
> which libreboot is a downstream) with and without BLS support, if versions
> that support it exist.

Yes, I think that we should just disable for all coreboot variants even if there is one that supports BLS. That's what we also do for Xen DomU (where some variants do support BLS) and ppc64le (even when only some old OPAL firmware don't support it).

Comment 10 Javier Martinez Canillas 2019-10-15 10:27:13 UTC
Created attachment 1625939 [details]
99-grub-mkconfig.install

Could you please copy the attached file to /usr/lib/kernel/install.d/ and give a try if it solves your issue?

After copying the file, try to reinstall your kernel with 'dnf reinstall kernel-core' and check if the /boot/grub2/grub.cfg has been properly generated.

This should also set GRUB_ENABLE_BLSCFG=false in /etc/default/grub so running grub2-mkconfig -o /boot/grub2/grub.cfg should work (for example if you want to change the GRUB_CMDLINE_LINUX in /etc/default/grub).

Comment 11 Javier Martinez Canillas 2019-10-15 12:23:15 UTC
Created attachment 1625975 [details]
99-grub-mkconfig.install

Attached new version that fixed a typo in the bios_vendor sysfs path.

Comment 12 Steven Haigh 2019-10-15 12:43:14 UTC
For what its worth, I have a PC Engines APU2 - this boots fine with BLS enabled.

BIOS Information
        Vendor: coreboot
        Version: v4.9.0.1
        Release Date: 01/09/2019
        ROM Size: 8192 kB
        Characteristics:
                PCI is supported
                PC Card (PCMCIA) is supported
                BIOS is upgradeable
                Selectable boot is supported
                ACPI is supported
                Targeted content distribution is supported
        BIOS Revision: 4.0
        Firmware Revision: 0.0

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: PC Engines
        Product Name: apu2
        Version: 1.0
        Serial Number: X
        UUID: Not Settable
        Wake-up Type: Reserved
        SKU Number: 4 GB
        Family: Not Specified

BIOS details:
    https://pcengines.github.io/

Comment 13 Demetris M 2019-10-15 13:08:04 UTC
(In reply to Javier Martinez Canillas from comment #11)
> Created attachment 1625975 [details]
> 99-grub-mkconfig.install
> 
> Attached new version that fixed a typo in the bios_vendor sysfs path.

Your  fix appears to be working, Javier, but there's something I didn't expect: 

I removed the line GRUB_ENABLE_BLSCFG="false" I previously put in my /etc/default/grub. I copied your 99-grub-mkconfig.install and reinstalled the kernel. The new grub.cfg is functionally identical to my previous one and the system boots (it has changes in spaces and some boot arguments that don't seem to make a difference). Same test with grub2-mkconfig, same minor differences, but no boot issues.

Yet, the line GRUB_ENABLE_BLSCFG=false didn't show up in the /etc/default/grub. I'm not sure I'm reading it correctly, but does the regex only match "=true" and changes into false? So if the line is not there at all (which I think was how that file was when I first upgraded to this release), nothing happens. So I'm not sure if the fix work, or something else was also disabling BLS elsewhere and that's why the new cfg was not using BLS.


To test, I set it to "true" and retested. After reinstallation, the regex works, but dnf gives this error 

----
Running transaction
   Running scriptlet: kernel-core-5.2.18-200.fc30.x86_64                                                                                                                         2/2 
grubby fatal error: unable to find a suitable template
grubby: doing this would leave no kernel entries. Not writing out new config.
---

Despite that, there was a new grub.cfg written, and it uses the old standard (didn't reboot to make sure).

If I change GRUB_ENABLE_BLSCFG=false to GRUB_ENABLE_BLSCFG="false" (with quotes) then the error goes away, but still, can you tell if that worked as intended?

Comment 14 Javier Martinez Canillas 2019-10-16 09:21:34 UTC
(In reply to mdtha from comment #13)
> (In reply to Javier Martinez Canillas from comment #11)
> > Created attachment 1625975 [details]
> > 99-grub-mkconfig.install
> > 
> > Attached new version that fixed a typo in the bios_vendor sysfs path.
> 
> Your  fix appears to be working, Javier, but there's something I didn't
> expect: 
> 
> I removed the line GRUB_ENABLE_BLSCFG="false" I previously put in my
> /etc/default/grub. I copied your 99-grub-mkconfig.install and reinstalled
> the kernel. The new grub.cfg is functionally identical to my previous one
> and the system boots (it has changes in spaces and some boot arguments that

Yes, I guess that you have the grubby-deprecated package installed? If that's the case, then this package will be used to add / remove menu entries from grub.cfg if you don't have GRUB_ENABLE_BLSCFG set to true.

If you remove the grubby-deprecated package, then grub2-mkconfig will be used to re-generate the grub.cfg file.

> don't seem to make a difference). Same test with grub2-mkconfig, same minor
> differences, but no boot issues.
> 
> Yet, the line GRUB_ENABLE_BLSCFG=false didn't show up in the
> /etc/default/grub. I'm not sure I'm reading it correctly, but does the regex
> only match "=true" and changes into false? So if the line is not there at
> all (which I think was how that file was when I first upgraded to this

That's correct, it just sets it to false if is not set to true. Since not having that set is equivalent to have it to false. In other words, it's only taken into account if GRUB_ENABLE_BLSCFG is true.

> release), nothing happens. So I'm not sure if the fix work, or something
> else was also disabling BLS elsewhere and that's why the new cfg was not
> using BLS.
> 
> 
> To test, I set it to "true" and retested. After reinstallation, the regex
> works, but dnf gives this error 
> 
> ----
> Running transaction
>    Running scriptlet: kernel-core-5.2.18-200.fc30.x86_64                    
> 2/2 
> grubby fatal error: unable to find a suitable template
> grubby: doing this would leave no kernel entries. Not writing out new config.
> ---
>

This is the old grubby tool (from grubby-deprecated) trying to edit the grub.cfg but failing for some reasons. I would just remove that tool and let grub2-mkconfig generate the grub.cfg in your case. This package is going away soon anyways, it was still left for backward compatibility for the cases where BLS couldn't be used as is your case. But I think that for those cases we should just run grub2-mkconfig and deprecated grubby.

> Despite that, there was a new grub.cfg written, and it uses the old standard
> (didn't reboot to make sure).
> 
> If I change GRUB_ENABLE_BLSCFG=false to GRUB_ENABLE_BLSCFG="false" (with
> quotes) then the error goes away, but still, can you tell if that worked as
> intended?

That's weird. It shouldn't make a difference.

Comment 15 Javier Martinez Canillas 2019-10-16 09:36:50 UTC
(In reply to mdtha from comment #13)
> (In reply to Javier Martinez Canillas from comment #11)
> > Created attachment 1625975 [details]
> > 99-grub-mkconfig.install
> > 
> > Attached new version that fixed a typo in the bios_vendor sysfs path.
> 
> Your  fix appears to be working, Javier, but there's something I didn't
> expect: 
> 

One problem is that I don't think we can deploy this fix. Because it assumes that all Coreboot machines are not able to boot BLS but that's not the case. Coreboot can have different payloads (SeaBIOS, TianoCore, GRUB, etc) and is only in the case of GRUB being a payload that BLS is not supported.

As mentioned by Steven in Comment 12, he uses Coreboot + SeaBIOS and could use the BLS configuration on a legacy BIOS install. The same happen with users that have a machine with Coreboot + TianoCore.

One option is to use /sys/class/dmi/id/board_* or /sys/class/dmi/id/product_* instead of /sys/class/dmi/id/bios_vendor and check for machines that comes with Libreboot, but that will lead to an unmaintanable mess.

Would it be possible for Libreboot to add a -libreboot suffix or something to the /sys/class/dmi/id/bios_version so we can distinguish between Libreboot and other Coreboots bootloaders?

Comment 16 Ben Cotton 2019-10-31 18:45:47 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Ben Cotton 2019-11-27 23:33:29 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.