Bug 2203203

Hi Renaud,

as you know, starting in RHEL-9, adding boot parameters and making similar changes should be done using grubby. :) I know that not all options (like GRUB_TIMEOUT) can be changed / added using grubby, but BLS files should not be edited by hand. 
# grubby --update-kernel /boot/vmlinuz-5.14.0-162.6.1.el9_1.x86_64 --args "CUSTOMARG"
will achieve what the customer is trying to do.

As for why there is mention of kernelopts in the grub.cfg file even though kernelopts to hold boot params got deprecated... I suspect this is just left over from RHEL-8 and it's just a variable name. It could get changed, but I don't think it's doing harm, per se.

Hi Marta,

OK fine, but how to deal with the following scenario:

I want to duplicate a BLS entry to have 2 snippets, one without CUSTOM arg, one with (e.g. a boot entry with "rhgb quiet" and one without for troubleshooting).
Using `grubby --update-kernel /boot/...` won't work since it will update both the BLS snippets, due to being only able to select an entry for a given "kernel", and not a given BLS snippet.


Additionally, since grubby doesn't support changing Grub timeout, Grub terminal (serial console, etc.) and other stuff like that, how can be achieve this since "grub2-mkconfig" just breaks all BLS snippets now?

# cp /boot/vmlinuz-5.14.0-162.6.1.el9_1.x86_64 /boot/custom_kernel
# cp /boot/initramfs-5.14.0-162.6.1.el9_1.x86_64.img /boot/custom_initramfs
# grubby --add-kernel /boot/custom_kernel --initrd /boot/custom_initramfs --title "Custom kernel" --copy-default --args "CUSTOM_ARG"

BUT, you're right that the options are getting clobbered after running grub2-mkconfig. Although based on my experiments, not on every machine..? Anyway, that's definitely a bug.

The reason that the bls conf file is getting clobbered is that the 10_linux script has GRUBENV_UPDATE set to yes by default. So, if you run
# grub2-mkconfig -o /path/to/grub.cfg
it will change the bls snippets to reflect what's in GRUB_CMDLINE_LINUX in /etc/default/grub

Running the commands in comment#3 followed by
# grub2-mkconfig -o /path/to/grub.cfg --no-grubenv-update
will generate a new grub.cfg with whatever potential changes made to /etc/default/grub (like TIMEOUT, SERIAL_COMMAND, etc.) as well as changes to .conf files made (either by hand--discouraged-- or using grubby).
I guess this is a change of behavior...? but it seems to work.

Created attachment 1974472 [details]
patch

Here is a patch that should fix this in a reasonable way. We are still discussing this behavior wrt another bug (bz#2212320).
The patch makes it so that if GRUB_ENABLE_BLSCFG is true or missing, the command line in BLS snippets does not get overwritten by GRUB_CMDLINE_LINUX in /etc/default/grub even if --no-grubenv-update is not passed. In order to overwrite, one needs to pass --update-bls-cmdline. This will require documentation...

Created attachment 1976267 [details]
patch v2

fix patch so that --update-bls-cmdline argument actually works and is mentioned in 'usage'

Reproduced with grub2-2.06-65.el9 in RHEL-9.3.0-20230725.27

# grubby --update-kernel /boot/vmlinuz-5.14.0-344.el9.ppc64le --args "args4u"

# cat /etc/default/grub 
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="ofconsole"
GRUB_CMDLINE_LINUX="crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINFO="terminfo -g 80x24 console"
GRUB_DISABLE_OS_PROBER=true

# grub2-mkconfig -o /etc/grub2.cfg
# grubby --info /boot/vmlinuz-5.14.0-344.el9.ppc64le 
index=0
kernel="/boot/vmlinuz-5.14.0-344.el9.ppc64le"
args="ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub"
root="/dev/mapper/rhel_ibm--p9z--26--lp12-root"
initrd="/boot/initramfs-5.14.0-344.el9.ppc64le.img"
title="Red Hat Enterprise Linux (5.14.0-344.el9.ppc64le) 9.3 (Plow)"
id="89f7902ad50a42f89ad85141cfaa3612-5.14.0-344.el9.ppc64le"

(BLS cmdline gets overwritten)

rebooting

# cat /proc/cmdline 
BOOT_IMAGE=(ieee1275//vdevice/v-scsi@30000003/disk@8100000000000000,msdos2)/vmlinuz-5.14.0-344.el9.ppc64le root=/dev/mapper/rhel_ibm--p9z--26--lp12-root ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub


Tested using grub2-2.06-67.el9 using the same steps and BLS snippets are not overwritten.

rebooting:
# cat /proc/cmdline 
BOOT_IMAGE=(ieee1275//vdevice/v-scsi@30000003/disk@8100000000000000,msdos2)/vmlinuz-5.14.0-344.el9.ppc64le root=/dev/mapper/rhel_ibm--p9z--26--lp12-root ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap args4u

# grub2-mkconfig -o /etc/grub2.cfg --update-bls-cmdline
# grubby --info /boot/vmlinuz-5.14.0-344.el9.ppc64le 
index=0
kernel="/boot/vmlinuz-5.14.0-344.el9.ppc64le"
args="ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub"
root="/dev/mapper/rhel_ibm--p9z--26--lp12-root"
initrd="/boot/initramfs-5.14.0-344.el9.ppc64le.img"
title="Red Hat Enterprise Linux (5.14.0-344.el9.ppc64le) 9.3 (Plow)"
id="89f7902ad50a42f89ad85141cfaa3612-5.14.0-344.el9.ppc64le"

If GRUB_ENABLE_BLSCFG=false then BLS snippets are also not overwritten, but rebooting gets args from /etc/default/grub to kernel:
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-5.14.0-344.el9.ppc64le root=/dev/mapper/rhel_ibm--p9z--26--lp12-root ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub

This change breaks existing automation without any backwards compatibility. In earlier versions of grub2-mkconfig, there's no such parameter. One would have to figure out the parameters to pass to grub2-mkconfig based on package version?! Basic commands should remain consistent in a major RHEL/CentOS version imo. 
Further, this also introduces breaking changes in regard to previous behaviour. If one updates /etc/grub/default with new kernel parameters and re-runs grub2-mkconfig, it will fail to generate the BLS config files (as this now requires the explicit switch). Please review this change carefully, as it should not break existing, established behaviour.

Created attachment 1982355 [details]
patch v3

Third time's the charm

Previous version of this patch broke installation on BIOS and PPC. Sorry about that! 
Current version fixes the bug, and does not break installation on BIOS or PPC and installation still works on UEFI too. Private comment with links to jobs: https://bugzilla.redhat.com/show_bug.cgi?id=2212320#c43

(In reply to Andrei Neagoe from comment #14)
> This change breaks existing automation without any backwards compatibility.
> In earlier versions of grub2-mkconfig, there's no such parameter. One would
> have to figure out the parameters to pass to grub2-mkconfig based on package
> version?! Basic commands should remain consistent in a major RHEL/CentOS
> version imo. 
> Further, this also introduces breaking changes in regard to previous
> behaviour. If one updates /etc/grub/default with new kernel parameters and
> re-runs grub2-mkconfig, it will fail to generate the BLS config files (as
> this now requires the explicit switch). Please review this change carefully,
> as it should not break existing, established behaviour.

Hi,
You make a valid point, but it's at odds with the bug description in general... You're right that default behavior is changing. On the other hand, BLS is what we want people to use, and they should also be using grubby to update cmdline arguments rather than making changes in /etc/default/grub
If you have multiple kernels installed with different cmdline arguments, those should not get overwritten.

If you set GRUB_ENABLE_BLSCFG=false, then BLS snippets will be unaffected and grub2-mkconfig will work as you expect.

Reproduced with grub2-2.06-68.el9: grub2-mkconfig updates BLS cmdline by default 

[root@sweetpig-9 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial"
GRUB_CMDLINE_LINUX="console=tty0 elevator=noop crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_sweetpig--9-swap rd.lvm.lv=rhel_sweetpig-9/root rd.lvm.lv=rhel_sweetpig-9/swap console=ttyS0 etc_def_grub"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@sweetpig-9 ~]# grubby --update-kernel /boot/vmlinuz-5.14.0-350.el9.x86_64 --args "some_args"
[root@sweetpig-9 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro console=tty0 elevator=noop crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_sweetpig--9-swap rd.lvm.lv=rhel_sweetpig-9/root rd.lvm.lv=rhel_sweetpig-9/swap console=ttyS0 some_args"
root="/dev/mapper/rhel_sweetpig--9-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="9cc32a47feed477e95d52efa8e78b0b0-5.14.0-350.el9.x86_64"

[root@sweetpig-9 ~]# grub2-mkconfig -o /etc/grub2.cfg 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

[root@sweetpig-9 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro console=tty0 elevator=noop crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_sweetpig--9-swap rd.lvm.lv=rhel_sweetpig-9/root rd.lvm.lv=rhel_sweetpig-9/swap console=ttyS0 etc_def_grub"
root="/dev/mapper/rhel_sweetpig--9-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="9cc32a47feed477e95d52efa8e78b0b0-5.14.0-350.el9.x86_64"


Verified (on x86, ppc64le, aarch64) with grub2-2.06-70.el9_3: BLS cmdline only updated (correctly) in the installer environment and upon user request -- not by default
[root@kvm-04-guest21 ~]# grep CMDLINE /etc/default/grub 
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200"

[root@kvm-04-guest21 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-350.el9.x86_64 root=/dev/mapper/rhel_kvm--04--guest21-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200

[root@kvm-04-guest21 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_kvm--04--guest21-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 

[root@kvm-04-guest21 ~]# grubby --update-kernel /boot/vmlinuz-5.14.0-350.el9.x86_64 --args "some_args"
[root@kvm-04-guest21 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 some_args"
root="/dev/mapper/rhel_kvm--04--guest21-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="65d1391391aa4d62bc35c50923807afb-5.14.0-350.el9.x86_64"
 
[root@kvm-04-guest21 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 etc_def_grub"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@kvm-04-guest21 ~]# grub2-mkconfig -o /etc/grub2.cfg 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

[root@kvm-04-guest21 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 some_args"
root="/dev/mapper/rhel_kvm--04--guest21-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="65d1391391aa4d62bc35c50923807afb-5.14.0-350.el9.x86_64"

[root@kvm-04-guest21 ~]# grub2-mkconfig -o /etc/grub2.cfg --update-bls-cmdline
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

[root@kvm-04-guest21 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 etc_def_grub"
root="/dev/mapper/rhel_kvm--04--guest21-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="65d1391391aa4d62bc35c50923807afb-5.14.0-350.el9.x86_64"

Re-built with correct tag, but same content

$ diff -q grub-9_3 grub-9_3.1
Only in grub-9_3.1: grub2-2.06-70.el9_3.1.src.rpm
Only in grub-9_3: grub2-2.06-70.el9_3.src.rpm
Files grub-9_3/grub2.spec and grub-9_3.1/grub2.spec differ

$ diff grub-9_3/grub2.spec grub-9_3.1/grub2.spec
19c19
< Release:	70%{?dist}
---
> Release:	70%{?dist}.1
535a536,541
> * Thu Sep 7 2023 Nicolas Frayer <nfrayer> - 2.06-70.el9_3.1
> - Bump spec release version
> - Related: #2203203
> - Related: #2212320
> - Related: #2221543
> 

gating tests grub2 / etc_default_grub / kernel_cmdline passed, so it is working as expected.

[root@kvm-07-guest39 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--07--guest39-swap rd.lvm.lv=rhel_kvm-07-guest39/root rd.lvm.lv=rhel_kvm-07-guest39/swap console=ttyS0,115200"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@kvm-07-guest39 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-362.el9.x86_64 root=/dev/mapper/rhel_kvm--07--guest39-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--07--guest39-swap rd.lvm.lv=rhel_kvm-07-guest39/root rd.lvm.lv=rhel_kvm-07-guest39/swap console=ttyS0,115200

grub2-2.06-70.el9_3.1 is in RHEL-9.3.0-20230908.62 nightly, on which we ran our Tier2 automated + manual tests
Moving to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (grub2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6653