The way that the kernel command line is integrated with the Boot Loader Specification changed in RHEL 9. As you noted, previously "$kernelopts" was included, which was set in the grubenv file. Now, to get static BLS config files, we include the kernel command line options when the BLS snippets are generated via "20-grub.install" and the options are taken from /etc/kernel/cmdline. In order to modify the kernel parameters permanently one has to edit that file.

Frank, what tooling did you use to change the kernel command line parameters? Maybe that tooling needs an update.

Thanks, Christian

I used grubby tool to update kernel boot parameters.
Here are results after appended 'kmemleak=on' as test parameter, grubby did not update "/etc/kernel/cmdline", so the new installed kernel does not carry over my test parameter.
I am moving this bug to grubby for more discussion.

The status after boot up rhel9 guest image.
[root@localhost ~]# grubby --update-kernel=ALL --args='kmemleak=on'
[root@localhost ~]# reboot
[root@localhost ~]# cat /etc/kernel/cmdline
root=UUID=95328833-60be-4de1-af21-8ac2d7dc6a37 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto
[root@localhost ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/boot/vmlinuz-5.14.0-1.2.1.el9.x86_64 root=UUID=95328833-60be-4de1-af21-8ac2d7dc6a37 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M kmemleak=on

After installed new kernel and reboot, 'kmemleak=one' is not carried over.
[root@localhost ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/boot/vmlinuz-5.14.0-4.el9.x86_64 root=UUID=95328833-60be-4de1-af21-8ac2d7dc6a37 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crash
kernel=auto

If I'm understanding correctly, and Christian Kellner mentions this too in c#2, this is because RHEL 9 doesn't use $kernelopts anymore. Here is a response from Javier Martinez Canillas to a similar question recently:
"""
> Just so I'm clear, you're saying that kernelopts is not set in RHEL 9?
>

That is correct, it's not used anymore in neither Fedora nor RHEL 9. It is
still in RHEL 8 since we couldn't drop it in the middle of a RHEL life cycle.

Using an environment variable in GRUB's env block turned out to cause more
harm than good, so it was decided to be dropped and instead store the cmdline
in the BLS snippets themselves. You can get a better explanation from:

https://src.fedoraproject.org/rpms/grub2/c/4a742183a39f

Doing that way also aligns better with what the OSTree-based Fedora/RHEL OS
variants do and also what's done for other arches (i.e: the zipl bootloader
used in s390x doesn't support vars and the cmdline is stored in BLS files).
"""

Hi Tomas,

Could you help check if '/etc/kernel/cmdline' is generated by imagebuilder?
There is no such file if system is installed from iso directly and new installed kernel always picks current boot options from /proc/cmdline as expected.

In this case, if '/etc/kernel/cmdline' exists, the new installed kernel picks it with higher priority and ignore all I appended in current running kernel.
I guess that most users want to take '/proc/cmdline' to new installed kernel instead of restoring it to default in "/etc/kernel/cmdline".

# cat /etc/kernel/cmdline
root=UUID=8d347296-a296-47f6-b707-e1c9aac9b833 console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto

Below is instruction from systemd.
<listitem>
    <para>Read by <filename>90-loaderentry.install</filename>. The content of the file
    <filename>/etc/kernel/cmdline</filename> specifies the kernel command line to use.  If that file does not
    exist, <filename>/proc/cmdline</filename> is used.</para>
</listitem>

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-31.el9.x86_64 root=UUID=8d347296-a296-47f6-b707-e1c9aac9b833 console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

Hi Frank,

in the KVM guest image (and indeed all guest/cloud images that were build via Image Builder) we always had a configuration file for the kernel command line: For RHEL8 (and Fedora < 33) it was the grubenv file in /boot/grub2/grubenv which contained the "kernelopts" variable. The Boot Loader Specification snippets that were generated during kernel install by 20-grub.install (provided by grub2) always had "options \$kernelopts", i.e. all the kernel options came only from the grubenv file.
This changed in Fedora 33 (see [1]). Now the Boot Loader Specification snippets always contain the full command line (grubenv is not used anymore). Instead, as you pointed out, /usr/kernel/cmdline, then /etc/kernel/cmdline is used for the system-wide persistent configuration and if neither exist we fall back to /proc/cmdline. Image Builder followed that change and now instead of writing the persistent kernel command line to /boot/grub2/grubenv it is writing it to /etc/kernel/cmdline.

Not writing the persistent kernel command line to a configuration file would be a change in behaviour, that could easily lead to unexpected behaviour. One example: you boot the system but change the kernel command line via the grub2 menu, so that for the current kernel it is changed (but only transiently, i.e. rebooting the system would give you the old kernel command line). If you install a new kernel while you have that transient change booted and we have no configuration file, that transient command line would now (via /proc/cmdline) end up in the newly installed kernel, but _not_ in the old kernel.

This is all a long version of saying, it is intended to have a persistent kernel configuration which used to be /boot/grub2/grubenv and now is /usr/kernel/cmdline.


[1] https://src.fedoraproject.org/rpms/grub2/c/4a742183a39f344a7685bccdc76d5e64dea3766a?branch=f33

(In reply to Christian Kellner from comment #8)
> Hi Frank,
> 
> in the KVM guest image (and indeed all guest/cloud images that were build
> via Image Builder) we always had a configuration file for the kernel command
> line: For RHEL8 (and Fedora < 33) it was the grubenv file in
> /boot/grub2/grubenv which contained the "kernelopts" variable. The Boot
> Loader Specification snippets that were generated during kernel install by
> 20-grub.install (provided by grub2) always had "options \$kernelopts", i.e.
> all the kernel options came only from the grubenv file.
> This changed in Fedora 33 (see [1]). Now the Boot Loader Specification
> snippets always contain the full command line (grubenv is not used anymore).
> Instead, as you pointed out, /usr/kernel/cmdline, then /etc/kernel/cmdline
> is used for the system-wide persistent configuration and if neither exist we
> fall back to /proc/cmdline. Image Builder followed that change and now
> instead of writing the persistent kernel command line to /boot/grub2/grubenv
> it is writing it to /etc/kernel/cmdline.
> 
> Not writing the persistent kernel command line to a configuration file would
> be a change in behaviour, that could easily lead to unexpected behaviour.
> One example: you boot the system but change the kernel command line via the
> grub2 menu, so that for the current kernel it is changed (but only
> transiently, i.e. rebooting the system would give you the old kernel command
> line). If you install a new kernel while you have that transient change
> booted and we have no configuration file, that transient command line would
> now (via /proc/cmdline) end up in the newly installed kernel, but _not_ in
> the old kernel.
> 
> This is all a long version of saying, it is intended to have a persistent
> kernel configuration which used to be /boot/grub2/grubenv and now is
> /usr/kernel/cmdline.
> 
> 
> [1]
> https://src.fedoraproject.org/rpms/grub2/c/
> 4a742183a39f344a7685bccdc76d5e64dea3766a?branch=f33

Thanks, Christian

So I guess grubby tool should handle such case accordingly when update, add, remove boot parameters, make changes in '/etc/kernel/cmdline' if it exists.

*** This bug has been marked as a duplicate of bug 1969362 ***