Bug 2059776 - Bootloader install fails on ostree installs since Fedora-Rawhide-20220301.n.0
Summary: Bootloader install fails on ostree installs since Fedora-Rawhide-20220301.n.0
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: 36
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Javier Martinez Canillas
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-01 23:27 UTC by Adam Williamson
Modified: 2022-03-26 15:18 UTC (History)
14 users (show)

Fixed In Version: grub2-2.06-22.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-25 19:33:52 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2022-03-01 23:27:17 UTC
Since Fedora-Rawhide-20220301.n.0 , when installing from an ostree installer image (e.g. the Silverblue installer , https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220301.n.0/compose/Silverblue/x86_64/iso/Fedora-Silverblue-ostree-x86_64-Rawhide-20220301.n.0.iso ), bootloader installation fails. A dialog saying "failed to write boot loader configuration" is shown. In the system logs, we see this:

09:20:57,511 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.modules.storage.bootloader.utils:Installing the boot loader.
09:20:57,511 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.modules.storage.bootloader.utils:boot loader stage1 target device is vda
09:20:57,511 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.modules.storage.bootloader.utils:boot loader stage2 target device is vda1
09:20:58,498 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.modules.storage.bootloader.grub2:bootloader.py: mbr will be updated for grub2
09:20:58,499 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running in chroot '/mnt/sysroot'... grub2-install --no-floppy /dev/vda
09:20:59,534 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Installing for i386-pc platform.
09:20:59,535 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Installation finished. No error reported.
09:20:59,536 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:program:Return code: 0
09:20:59,555 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.modules.storage.bootloader.grub2:bootloader.py: used boot args: rhgb quiet
09:20:59,555 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running in chroot '/mnt/sysroot'... grub2-editenv - set menu_auto_hide=1 boot_success=1
09:20:59,565 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:program:Return code: 0
09:20:59,565 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
09:21:02,377 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Generating grub configuration file ...
09:21:02,378 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:error: loading sysroot: No ostree= kernel argument found
09:21:02,378 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:program:Return code: 1
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:ERROR:anaconda.modules.storage.bootloader.installation:Bootloader installation has failed: failed to write boot loader configuration
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:Traceback (most recent call last):
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/installation.py", line 139, in run
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:    install_boot_loader(storage=self._storage)
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/utils.py", line 215, in install_boot_loader
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:    storage.bootloader.write()
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/grub2.py", line 456, in write
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self.write_config()
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/grub2.py", line 373, in write_config
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:    raise BootLoaderError("failed to write boot loader configuration")
09:21:02,388 WARNING org.fedoraproject.Anaconda.Modules.Storage:pyanaconda.modules.storage.bootloader.base.BootLoaderError: failed to write boot loader configuration

I suspect this is a consequence of the "Skip machine ID check when updating BLS" change - see https://bugzilla.redhat.com/show_bug.cgi?id=2032680 for more on that. See https://bugzilla.redhat.com/show_bug.cgi?id=1753485 , specifically https://bugzilla.redhat.com/show_bug.cgi?id=1753485#c20 , where Davide saw the same error during testing of an earlier issue in this area. I suspect that because we threw away the machine ID checks in 10_linux, we're now trying to update things on ostree deployments that we can't really update, or something like that.

grub2 is the only related package that seems to have changed in the 20220301.n.0 compose. anaconda didn't, and nothing ostree related did.

Comment 1 Adam Williamson 2022-03-01 23:31:32 UTC
Proposing as an FE for Beta, obviously we don't want to ship Silverblue installer images that can't install a bootloader.

Comment 2 Javier Martinez Canillas 2022-03-02 08:26:24 UTC
(In reply to Adam Williamson from comment #0)
> Since Fedora-Rawhide-20220301.n.0 , when installing from an ostree installer

[snip]

> 
> I suspect this is a consequence of the "Skip machine ID check when updating
> BLS" change - see https://bugzilla.redhat.com/show_bug.cgi?id=2032680 for
> more on that. See https://bugzilla.redhat.com/show_bug.cgi?id=1753485 ,
> specifically https://bugzilla.redhat.com/show_bug.cgi?id=1753485#c20 , where
> Davide saw the same error during testing of an earlier issue in this area. I
> suspect that because we threw away the machine ID checks in 10_linux, we're
> now trying to update things on ostree deployments that we can't really
> update, or something like that.
> 

I believe this analysis from Adam is spot on.

Probably 10_linux should test -d /ostree/repo or something (don't know what is
the most accurate way to check if is an OSTree installation) and don't attempt
to update the BLS snippets if that's the case.

Since for OSTree systems the BLS snippets belong to OSTree and not to grub2.

Comment 3 Colin Walters 2022-03-02 12:38:30 UTC
What we actually want is to stop using ostree-grub2 at all.  This relates to 
https://github.com/rhinstaller/anaconda/pull/2752

Fedora CoreOS has been set up this way for a long time now.

> Probably 10_linux should test -d /ostree/repo or something (don't know what is
the most accurate way to check if is an OSTree installation) and don't attempt
to update the BLS snippets if that's the case.

test -f /run/ostree-booted is most canonical.

Comment 4 Adam Williamson 2022-03-07 18:41:12 UTC
Seems the new grub2 was intentionally not built for F36 yet, so this doesn't need to be an FE for F36. Unproposing.

Comment 5 Fedora Update System 2022-03-10 20:18:58 UTC
FEDORA-2022-3886259a14 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3886259a14

Comment 6 Fedora Update System 2022-03-11 19:25:57 UTC
FEDORA-2022-3886259a14 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3886259a14`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3886259a14

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Adam Williamson 2022-03-15 01:12:33 UTC
The new grub2 doesn't seem to have fixed this, AFAICT. Test failed again in today's Rawhide, which should include that grub2. I verified that the ostree build task succeeded and included grub2-2.06-22.fc37 - https://koji.fedoraproject.org/koji/taskinfo?taskID=84147420 - but the install still fails and we see the same errors in the logs:

10:51:31,893 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running in chroot '/mnt/sysroot'... grub2-install --no-floppy /dev/vda
10:51:37,137 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Installing for i386-pc platform.
10:51:37,137 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Installation finished. No error reported.
10:51:37,137 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:program:Return code: 0
10:51:37,187 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.modules.storage.bootloader.grub2:bootloader.py: used boot args: rhgb quiet
10:51:37,190 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running in chroot '/mnt/sysroot'... grub2-editenv - set menu_auto_hide=1 boot_success=1
10:51:37,309 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:program:Return code: 0
10:51:37,312 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
10:51:45,629 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Generating grub configuration file ...
10:51:45,631 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:error: loading sysroot: No ostree= kernel argument found
10:51:45,631 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:program:Return code: 1
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:ERROR:anaconda.modules.storage.bootloader.installation:Bootloader installation has failed: failed to write boot loader configuration
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:Traceback (most recent call last):
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/installation.py", line 139, in run
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:    install_boot_loader(storage=self._storage)
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/utils.py", line 215, in install_boot_loader
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:    storage.bootloader.write()
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/grub2.py", line 456, in write
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self.write_config()
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/bootloader/grub2.py", line 373, in write_config
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:    raise BootLoaderError("failed to write boot loader configuration")
10:51:45,646 WARNING org.fedoraproject.Anaconda.Modules.Storage:pyanaconda.modules.storage.bootloader.base.BootLoaderError: failed to write boot loader configuration

Comment 8 Robbie Harwood 2022-03-15 16:49:00 UTC
Unfortunate.  I don't know why that would be then - the fix I applied looks to me like what was suggested in #c2 / #c3 : https://src.fedoraproject.org/rpms/grub2/c/e31fc7ca96eb797987d2e627220e71dec48f1152?branch=rawhide  Maybe Javier or Colin can provide more information here?

Comment 9 Adam Williamson 2022-03-22 22:18:37 UTC
Looking at this, I suspect the problem may be that during initial install from the OStree DVD installer images, we're running grub2-mkconfig in a chroot to the installed system. The installed system isn't booted. So I suspect /run/ostree-booted does not exist in that case?

Comment 10 Adam Williamson 2022-03-23 04:08:20 UTC
It's a little hard to test the fix, but I'm hoping this will be fixed by https://bodhi.fedoraproject.org/updates/FEDORA-2022-3dc9b2c6b3 , will confirm with the next compose. I added the /ostree/repo check as well as the /run/ostree-booted check.

Comment 11 Adam Williamson 2022-03-23 22:27:58 UTC
Dang. Nope. Still broken with my attempted fix. It's a bit hard to poke this, because it uses the grub bits from the installed system and I don't know how to mess with those when the installed system is an ostree. What I'd want to do is run the grub scripts via sh -x on their *first* run, so I can see what happens on that first run through. I guess I'll try disabling bootloader installation in anaconda then make my changes and run it manually and see what happens...

Comment 12 Adam Williamson 2022-03-23 23:45:56 UTC
Oh, man, I'm an idiot, I put an extra bracket in my hand-edit of the patch. Sigh.

Comment 13 Adam Williamson 2022-03-24 15:56:18 UTC
OK, we fixed that in -29, so I'm hoping this is really fixed now, but...ostrees failed to build in today's Rawhide, so the Silverblue installer image has the ostree from yesterday's compose in it, so it doesn't have the fix. jlebon says they'll fix the ostree build bug, so hopefully we'll get new ostrees in the next compose and this will finally be fixed...

Comment 14 Adam Williamson 2022-03-25 19:33:52 UTC
OK, this is indeed finally fixed now I got my brackets right.

Comment 15 Fedora Update System 2022-03-26 15:18:27 UTC
FEDORA-2022-3886259a14 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.