Bug 2021193

Summary:	grubby --update-kernel=ALL --args=... does not set the kernel CLI argument for kernel upgrades
Product:	Red Hat Enterprise Linux 9	Reporter:	Jan Pazdziora (Red Hat) <jpazdziora>
Component:	grubby	Assignee:	Bootloader engineering team <bootloader-eng-team>
Status:	CLOSED DUPLICATE	QA Contact:	Release Test Team <release-test-team-automation>
Severity:	unspecified	Docs Contact:
Priority:	urgent
Version:	9.0	CC:	fmartine, jaredz, jpazdziora, mreznik, podvody, pstodulk, rharwood, vpolasek
Target Milestone:	rc	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-18 16:54:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jan Pazdziora (Red Hat) 2021-11-08 14:20:05 UTC

Description of problem:

Based on https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9-beta/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel and bug 1926715 comment 3, the correct way of updating the kernel command line parameters for all kernels is using

   grubby --update-kernel=ALL --args="<NEW_PARAMETER>"

There is an implied assumption that this will affect not just the currently installed kernels but will set the system so that upgraded kernels get the new set of kernel command line parameters as well. At least, that's how RHEL 8.4 on x86_64 works.

However, on RHEL 9, this no longer works.

Version-Release number of selected component (if applicable):

grubby-8.40-54.el9.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. grep -r jezek= /boot /etc || :
2. grubby --update-kernel=ALL --args=jezek=8
3. grep -r jezek= /boot /etc
4. dnf reinstall -y kernel-core
5. grep -r jezek= /boot /etc
6. reboot
7. grep jezek=8 /proc/cmdline

Actual results:

1. no output

2. no output

3.
/boot/loader/entries/1991dc4333dd4d5da631ead543573ddc-5.14.0-11.el9.x86_64.conf:options root=/dev/mapper/rhel_cc--vm2p-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_cc--vm2p-swap rd.lvm.lv=rhel_cc-vm2p/root rd.lvm.lv=rhel_cc-vm2p/swap console=ttyS0,115200 jezek=8
/boot/loader/entries/1991dc4333dd4d5da631ead543573ddc-0-rescue.conf:options root=/dev/mapper/rhel_cc--vm2p-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_cc--vm2p-swap rd.lvm.lv=rhel_cc-vm2p/root rd.lvm.lv=rhel_cc-vm2p/swap console=ttyS0,115200 jezek=8
/etc/default/grub:GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_cc--vm2p-swap rd.lvm.lv=rhel_cc-vm2p/root rd.lvm.lv=rhel_cc-vm2p/swap console=ttyS0,115200 jezek=8"

4.
[...]
Running transaction
  Preparing        :                                                        1/1 
  Reinstalling     : kernel-core-5.14.0-11.el9.x86_64                       1/2 
  Running scriptlet: kernel-core-5.14.0-11.el9.x86_64                       1/2 
  Running scriptlet: kernel-core-5.14.0-11.el9.x86_64                       2/2 
  Cleanup          : kernel-core-5.14.0-11.el9.x86_64                       2/2 
  Running scriptlet: kernel-core-5.14.0-11.el9.x86_64                       2/2 
  Verifying        : kernel-core-5.14.0-11.el9.x86_64                       1/2 
  Verifying        : kernel-core-5.14.0-11.el9.x86_64                       2/2 
Installed products updated.

Reinstalled:
  kernel-core-5.14.0-11.el9.x86_64                                              

Complete!

5.
/boot/loader/entries/1991dc4333dd4d5da631ead543573ddc-0-rescue.conf:options root=/dev/mapper/rhel_cc--vm2p-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_cc--vm2p-swap rd.lvm.lv=rhel_cc-vm2p/root rd.lvm.lv=rhel_cc-vm2p/swap console=ttyS0,115200 jezek=8
/etc/default/grub:GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_cc--vm2p-swap rd.lvm.lv=rhel_cc-vm2p/root rd.lvm.lv=rhel_cc-vm2p/swap console=ttyS0,115200 jezek=8"

7. no output

So even if the value got set to GRUB_CMDLINE_LINUX, it was not used for kernel reinstall (and it does not work for upgrading to newer version either).

Expected results:

Output 5 the same as output 3.

7.
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-11.el9.x86_64 root=/dev/mapper/rhel_cc--vm2p-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_cc--vm2p-swap rd.lvm.lv=rhel_cc-vm2p/root rd.lvm.lv=rhel_cc-vm2p/swap console=ttyS0,115200 jezek=8

Additional info:

On RHEL 8 with grubby-8.40-41.el8.x86_64, things work because the value gets set to /boot/efi/EFI/redhat/grubenv and it get referenced via $kernelopts:

# grep options /boot/loader/entries/*.conf
/boot/loader/entries/8b08d51dd79c4ef8976aa5887ed47fc8-0-rescue.conf:options $kernelopts
/boot/loader/entries/8b08d51dd79c4ef8976aa5887ed47fc8-4.18.0-305.el8.x86_64.conf:options $kernelopts $tuned_params

At least on default RHEL 8.4 installation. Outputs 3 and 5 are
/boot/efi/EFI/redhat/grubenv:kernelopts=root=/dev/mapper/rhel_cc--vm3d-root ro crashkernel=auto resume=/dev/mapper/rhel_cc--vm3d-swap rd.lvm.lv=rhel_cc-vm3d/root rd.lvm.lv=rhel_cc-vm3d/swap console=ttyS0,115200 jezek=8
/etc/default/grub:GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel_cc--vm3d-swap rd.lvm.lv=rhel_cc-vm3d/root rd.lvm.lv=rhel_cc-vm3d/swap console=ttyS0,115200 jezek=8"

and output 7 is

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-305.el8.x86_64 root=/dev/mapper/rhel_cc--vm3d-root ro crashkernel=auto resume=/dev/mapper/rhel_cc--vm3d-swap rd.lvm.lv=rhel_cc-vm3d/root rd.lvm.lv=rhel_cc-vm3d/swap console=ttyS0,115200 jezek=8

Comment 2 Jan Pazdziora (Red Hat) 2021-11-08 14:58:15 UTC

On s390x with zipl, the problem is also present, except it does not work on RHEL 8 either, and the GRUB_CMDLINE_LINUX mechanism is not used. Filed as bug 2021207.

Comment 3 Jan Pazdziora (Red Hat) 2021-11-10 09:20:05 UTC

I've filed bug 2021806 to capture the expectation WRT kernel upgrades.

Comment 4 Jan Pazdziora (Red Hat) 2021-11-10 13:10:38 UTC

Edited the bugzilla title back as the issue is present on ppc64le as well, and I presume on all systems that use GRUB.

Comment 6 Michal Reznik 2022-02-01 12:54:28 UTC

Not sure if this is related but we are hitting similar problem after in-place upgrade from RHEL 8 to RHEL 9. In our case we run:

grubby --update-kernel=/boot/vmlinuz-5.14.0-1.el9.x86_64 --args=selinux=0 (for systems where SELinux was disabled in '/etc/selinux/config') 

but the update is not reflected. In our case there is a difference that the system has "GRUB_ENABLE_BLSCFG=false" in '/etc/default/grub'. However despite this, grubby seems to be doing the opposite and is updating '/boot/loader/entries/*.conf' only.

Comment 9 Javier Martinez Canillas 2022-02-09 07:00:38 UTC

The grub kernel-install script (https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/20-grub.install) look at the following paths to add the kernel cmdline to the BLS snippet:

/etc/kernel/cmdline
/usr/lib/kernel/cmdline
/proc/cmdline

These paths are documented in the kernel-install man page and is what others bootloaders kernel-install scripts also do (i.e: sd-boot).

But these files are never created by neither Anaconda on installation nor grubby on kernel cmdline modifications.

A RFE to add support for this to grub2 has already been filed in bug #1931912 and I mentioned in bug #1931912, Comment 6 that grubby also needs to support this.

Comment 10 Jared Dominguez 2022-03-17 20:19:08 UTC

From talking to the team, it sounds like this should essentially be a RHEL 9.0 copy of 1931912. Beyond that, I don't see capacity for further work on this except possibly updating documentation.

Comment 11 Robbie Harwood 2022-03-17 21:01:17 UTC

*** Bug 2021207 has been marked as a duplicate of this bug. ***

Comment 12 Robbie Harwood 2022-03-18 16:54:51 UTC


*** This bug has been marked as a duplicate of bug 1969362 ***