Bug 2323514

(In reply to Martin Pilarski from comment #0)
> The script "/usr/lib/kernel/install.d/92-tuned.install" puts the
> $tuned_initrd variable into the initrd line of all boot loader entries
> (except the rescue image entry).
>
This is intended.

> The variable is not compatible with systemd-boot & prevents boot of the
> system (Error preparing initrd: not found).
>
It's using grub environment to easily reconfigure boot arguments. Unfortunately, I don't know how to support this feature with systemd-boot. This will require discussion with the systemd guys.
 
> Because all entries are affected, the upgrade from 40 to 41 resulted in a
> broken system (rescue image didn't seem to work for a different reason).
>
Only with systemd-boot. I think systemd-boot should ignore it. But even in such case the TuneD feature will not work with systemd-boot.

> The system uses the ESP as /boot-partition, as described in
> https://systemd.io/BOOT/ (i.e. boot loader entries are under
> /boot/loader/entries/).
> 
> $tuned_params in the options line has no effect on the boot.
> 
It will have effect if you select some TuneD profile utilizing bootloader plugin and setting custom initrd overlays.

*** Bug 2323348 has been marked as a duplicate of this bug. ***

IMHO workaround is to add empty (untested):
/etc/kernel/install.d/92-tuned.install

Is there a way how to reliably detect that system is using systemd-boot instead of the grub? I will add workaround into 92-tuned.install.

Interesting, that nobody hit/reported it during F41 test days.

(In reply to Jaroslav Škarvada from comment #4)
> Is there a way how to reliably detect that system is using systemd-boot
> instead of the grub? I will add workaround into 92-tuned.install.
> 
Or maybe rather the other way round, as it's grub extensions, i.e. enable it in case grub is detected.

I've been redirected here https://github.com/redhat-performance/tuned/issues/704#issuecomment-2455170137, though I raised a slightly different (but closely connected) issue, which is that after _upgrading_ to Fedora 41, the loader/entries files use these variables, as discussed above, but grub.cfg is not automatically updated to set a value for them.

Empty values only appear in grub.cfg after we re-run grub2-mkconfig or if we install a TuneD profile which actually uses these values (e.g. with [bootloader] cmdline_...=... &/or initrd_add_img=...).

This doesn't stop anything booting, but it seems to go against a stable convention that grub variables are always defined if referred to, in a well-set-up system. (And makes it harder for any third party software which wants to parse loader/entries, and relies on this convention, to do its work.)

Once one of the above steps is run, TuneD also inserts the values for tuned_initrd and tuned_params into grubenv. I do wonder if this is wise (grubenv has very limited space and these variables can be long; it is not clear why they are needed in both places?), but it looks as if a quick fix could involve adding initial blank values in either place.

PoC PR:
https://github.com/redhat-performance/tuned/pull/705

(In reply to Mike Beaton from comment #6)
> I've been redirected here
> https://github.com/redhat-performance/tuned/issues/704#issuecomment-
> 2455170137, though I raised a slightly different (but closely connected)
> issue, which is that after _upgrading_ to Fedora 41, the loader/entries
> files use these variables, as discussed above, but grub.cfg is not
> automatically updated to set a value for them.
> 
> Empty values only appear in grub.cfg after we re-run grub2-mkconfig or if we
> install a TuneD profile which actually uses these values (e.g. with
> [bootloader] cmdline_...=... &/or initrd_add_img=...).
> 
> This doesn't stop anything booting, but it seems to go against a stable
> convention that grub variables are always defined if referred to, in a
> well-set-up system. (And makes it harder for any third party software which
> wants to parse loader/entries, and relies on this convention, to do its
> work.)
> 
> Once one of the above steps is run, TuneD also inserts the values for
> tuned_initrd and tuned_params into grubenv. I do wonder if this is wise
> (grubenv has very limited space and these variables can be long; it is not
> clear why they are needed in both places?), but it looks as if a quick fix
> could involve adding initial blank values in either place.

There was no related change on the TuneD side. I will try to reproduce it.

Regarding the grubenv, IMHO it has 1024 bytes limit and there can be other variables. Regardless the limit it's clean and usually safe solution - edit kernel/initrd parameters just by changing variables. But taking into account recent systemd-boot deployment, we will have to rewrite it, which will also resolve the grubenv limit problem.

(In reply to Jaroslav Škarvada from comment #3)
> IMHO workaround is to add empty (untested):
> /etc/kernel/install.d/92-tuned.install

This workaround worked for me.

The problem also doesn't occur if ESP partition is moved from /boot/efi to /efi as suggested by some guidelines.

FEDORA-2024-ec96d2567e (tuned-2.24.0-6.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-ec96d2567e

FEDORA-2024-e0e2d7c34a (tuned-2.24.0-6.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-e0e2d7c34a

FEDORA-2024-ec96d2567e (tuned-2.24.0-6.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

By the way, TuneD is definitely not the only software to use grub variables in /loader/entries. I work on a non-GRUB BLSpec compliant bootloader, and we definitely had to write heuristic processing to extract variable values from grub.cfg in order to get most Linuces to boot. Including defaultkernelopts - which grub blscfg applies if there are not other kernel options. But there are definitely other distros, which use other vars in kernel options.

The specific problem with $tuned_initrd is worse for sd-boot (and other BLSpec compliant loaders) than just having an unknown variable in options, because the kernel ignores unknown options (option `$myopt` will be ignored, but will probably boot - unless what is was meant to expand to is required).

Having $tuned_initrd in the initrd line is slightly non-obviously different and harder to ignore (intentionally, or by mistake):

Firstly, unknown values are not ignored here, they are treated as unloadable files.

Secondly, BLSpec does eeven not allow multiple initrd files in one line, as is being done here, there are meant to be in separate initrd lines. But grub+blscfg does allow this. So from sd-boot's point of view, the file is `initrd-version.img $tuned_initrd` (a file name with ' '  and '$' in it). Not that it is much better even if we allow that it means two files `initrd-version.img` and `$tuned_initrd`, because sd-boot does not know about `$tuned_initrd`.

I wonder if there is any scope for proposing a shared, simple-to-parse variable store, next to /loader/entries, which both BLSpec and grub+blscfg are aware of, but which is the only place which /loader/entries variables can come from, so that BLSpec when done independently if grub can also find and use these. (I am not sure how/where/whether to propose this...)

(In reply to Mike Beaton from comment #13)
> Secondly, BLSpec does eeven not allow multiple initrd files in one line, as
> is being done here, there are meant to be in separate initrd lines. But
> grub+blscfg does allow this. So from sd-boot's point of view, the file is
> `initrd-version.img $tuned_initrd` (a file name with ' '  and '$' in it).
> Not that it is much better even if we allow that it means two files
> `initrd-version.img` and `$tuned_initrd`, because sd-boot does not know
> about `$tuned_initrd`.
>
We implemented this feature long time ago into TuneD when BLS even didn't exist and later ported it to BLS and used grub as the reference implementation, because at that time there wasn't other working BLS option. Multiple initrd lines shouldn't be problem, we could switch to this format. It's question whether it can cope with initrd lines with empty path, but even in such case we could workaround it by e.g. empty initrd, strange, but it may work.

> I wonder if there is any scope for proposing a shared, simple-to-parse
> variable store, next to /loader/entries, which both BLSpec and grub+blscfg
> are aware of, but which is the only place which /loader/entries variables
> can come from, so that BLSpec when done independently if grub can also find
> and use these. (I am not sure how/where/whether to propose this...)

This would be great to have and it would make our life simpler. I think the best would be to get it into BLS specification, next revision. To be honest I thought it's already there or at least mentioned as optional extension, but unfortunately it is not.

FEDORA-2024-e0e2d7c34a has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-e0e2d7c34a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-e0e2d7c34a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2024-e0e2d7c34a (tuned-2.24.0-6.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.