Hi,
The change from 2.06-61 --> 2.06-63 should not affect the kernel cmdline, and indeed this does not happen for a 'regular' installation / upgrade, but I know the setup on AWS is a bit different (and we don't have access).

Could you please:
1.Launch an aws instance with build(RHEL-9.3.0-20230531.0) which has grub2-2.06-61.el9
2.Update grub2-2.06-61.el9 to grub2-2.06-63.el9
3.Reboot
(from the Description) and show results of
# cat /proc/cmdline
# cat /etc/kernel/cmdline
# cat /etc/default/grub
# cat /etc/sysconfig/kernel
# grubby --info DEFAULT

both before and after the grub2 update

(In reply to Marta Lewandowska from comment #1)
> Hi,
> The change from 2.06-61 --> 2.06-63 should not affect the kernel cmdline,
> and indeed this does not happen for a 'regular' installation / upgrade, but
> I know the setup on AWS is a bit different (and we don't have access).
> 
> Could you please:
> 1.Launch an aws instance with build(RHEL-9.3.0-20230531.0) which has
> grub2-2.06-61.el9
> 2.Update grub2-2.06-61.el9 to grub2-2.06-63.el9
> 3.Reboot
> (from the Description) and show results of
> # cat /proc/cmdline
> # cat /etc/kernel/cmdline
> # cat /etc/default/grub
> # cat /etc/sysconfig/kernel
> # grubby --info DEFAULT
> 
> both before and after the grub2 update

1.Before grub2 update:

$ rpm -qa|grep grub*
grub2-common-2.06-61.el9.noarch
grub2-pc-modules-2.06-61.el9.noarch
grub2-tools-minimal-2.06-61.el9.x86_64
grub2-tools-2.06-61.el9.x86_64
grubby-8.40-63.el9.x86_64
grub2-pc-2.06-61.el9.x86_64

$ sudo cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-319.el9.x86_64 root=UUID=05d4f3ac-430d-4855-9dce-f71566d46dce console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

$ sudo cat /etc/kernel/cmdline
root=UUID=05d4f3ac-430d-4855-9dce-f71566d46dce console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295

$ sudo cat /etc/default/grub
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295"
GRUB_TIMEOUT=0
GRUB_ENABLE_BLSCFG=true
GRUB_DEFAULT=saved

$ sudo cat /etc/sysconfig/kernel
DEFAULTKERNEL=kernel
UPDATEDEFAULT=yes

$ sudo grubby --info DEFAULT
index=1
kernel="/boot/vmlinuz-5.14.0-319.el9.x86_64"
args="console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M $tuned_params"
root="UUID=05d4f3ac-430d-4855-9dce-f71566d46dce"
initrd="/boot/initramfs-5.14.0-319.el9.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.14.0-319.el9.x86_64) 9.3 (Plow)"
id="2d81540e0ae846c89336178ac67e8f03-5.14.0-319.el9.x86_64"

2.After update grub2 but no reboot:

$ rpm -qa|grep grub*
grubby-8.40-63.el9.x86_64
grub2-common-2.06-63.el9.noarch
grub2-tools-minimal-2.06-63.el9.x86_64
grub2-tools-2.06-63.el9.x86_64
grub2-pc-modules-2.06-63.el9.noarch
grub2-efi-x64-2.06-63.el9.x86_64
grub2-pc-2.06-63.el9.x86_64
grub2-tools-extra-2.06-63.el9.x86_64
grub2-efi-aa64-modules-2.06-63.el9.noarch
grub2-efi-x64-cdboot-2.06-63.el9.x86_64
grub2-efi-x64-modules-2.06-63.el9.noarch
grub2-tools-efi-2.06-63.el9.x86_64


$ sudo cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-319.el9.x86_64 root=UUID=05d4f3ac-430d-4855-9dce-f71566d46dce console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

$ sudo cat /etc/kernel/cmdline
root=UUID=05d4f3ac-430d-4855-9dce-f71566d46dce ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295

$ sudo cat /etc/default/grub
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295"
GRUB_TIMEOUT=0
GRUB_ENABLE_BLSCFG=true
GRUB_DEFAULT=saved

$ sudo cat /etc/sysconfig/kernel
DEFAULTKERNEL=kernel
UPDATEDEFAULT=yes

$ sudo grubby --info DEFAULT
index=1
kernel="/boot/vmlinuz-5.14.0-319.el9.x86_64"
args="ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295"
root="UUID=05d4f3ac-430d-4855-9dce-f71566d46dce"
initrd="/boot/initramfs-5.14.0-319.el9.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.14.0-319.el9.x86_64) 9.3 (Plow)"
id="2d81540e0ae846c89336178ac67e8f03-5.14.0-319.el9.x86_64

3.After reboot
$ sudo cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-319.el9.x86_64 root=UUID=05d4f3ac-430d-4855-9dce-f71566d46dce ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295

$ sudo cat /etc/kernel/cmdline
root=UUID=05d4f3ac-430d-4855-9dce-f71566d46dce ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 

$ sudo cat /etc/default/grub
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295"
GRUB_TIMEOUT=0
GRUB_ENABLE_BLSCFG=true
GRUB_DEFAULT=saved

$ sudo cat /etc/sysconfig/kernel
DEFAULTKERNEL=kernel
UPDATEDEFAULT=yes

$ sudo grubby --info DEFAULT
index=1
kernel="/boot/vmlinuz-5.14.0-319.el9.x86_64"
args="ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295"
root="UUID=05d4f3ac-430d-4855-9dce-f71566d46dce"
initrd="/boot/initramfs-5.14.0-319.el9.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.14.0-319.el9.x86_64) 9.3 (Plow)"
id="2d81540e0ae846c89336178ac67e8f03-5.14.0-319.el9.x86_64"

Thank you very much.

I believe that this is happening not because of the new grub, but as a result of fixing https://bugzilla.redhat.com/show_bug.cgi?id=2162299
If you could please check that by launching an aws instance with RHEL-9.3.0-20230531.0 (grub2-2.06-61.el9)
and then just:
# grub2-mkconfig
and rebooting to see what's on the cmdline..?

The same files from a regular installation look like this (you'll notice crashkernel is in both /etc/kernel/cmdline and /etc/default/grub while it's missing from the AWS image even before the grub update):

[root@kvm-08-guest38 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-319.el9.x86_64 root=/dev/mapper/rhel_kvm--08--guest38-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--08--guest38-swap rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap console=ttyS0,115200

[root@kvm-08-guest38 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_kvm--08--guest38-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--08--guest38-swap rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap console=ttyS0,115200 

[root@kvm-08-guest38 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--08--guest38-swap rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap console=ttyS0,115200"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

The fix for 2162299 was to write the same contents to /etc/kernel/cmdline and /etc/default/grub, but it looks like crashkernel got left out.

(In reply to Marta Lewandowska from comment #3)
> Thank you very much.
> 
> I believe that this is happening not because of the new grub, but as a
> result of fixing Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2162299
> If you could please check that by launching an aws instance with
> RHEL-9.3.0-20230531.0 (grub2-2.06-61.el9)
> and then just:
> # grub2-mkconfig
> and rebooting to see what's on the cmdline..?
> 
> The same files from a regular installation look like this (you'll notice
> crashkernel is in both /etc/kernel/cmdline and /etc/default/grub while it's
> missing from the AWS image even before the grub update):
> 
> [root@kvm-08-guest38 ~]# cat /proc/cmdline 
> BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-319.el9.x86_64
> root=/dev/mapper/rhel_kvm--08--guest38-root ro
> crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
> resume=/dev/mapper/rhel_kvm--08--guest38-swap
> rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap
> console=ttyS0,115200
> 
> [root@kvm-08-guest38 ~]# cat /etc/kernel/cmdline 
> root=/dev/mapper/rhel_kvm--08--guest38-root ro
> crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
> resume=/dev/mapper/rhel_kvm--08--guest38-swap
> rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap
> console=ttyS0,115200 
> 
> [root@kvm-08-guest38 ~]# cat /etc/default/grub 
> GRUB_TIMEOUT=5
> GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
> GRUB_DEFAULT=saved
> GRUB_DISABLE_SUBMENU=true
> GRUB_TERMINAL="serial console"
> GRUB_SERIAL_COMMAND="serial --speed=115200"
> GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
> resume=/dev/mapper/rhel_kvm--08--guest38-swap
> rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap
> console=ttyS0,115200"
> GRUB_DISABLE_RECOVERY="true"
> GRUB_ENABLE_BLSCFG=true
> 
> The fix for 2162299 was to write the same contents to /etc/kernel/cmdline
> and /etc/default/grub, but it looks like crashkernel got left out.

Thanks for the investigation.
After run #grub2-mkconfig and reboot, the issue can be reproduced. 

Tomas, could you take a look from osbuild side?

(In reply to libhe from comment #4)
> (In reply to Marta Lewandowska from comment #3)
> > Thank you very much.
> > 
> > I believe that this is happening not because of the new grub, but as a
> > result of fixing Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2162299
> > If you could please check that by launching an aws instance with
> > RHEL-9.3.0-20230531.0 (grub2-2.06-61.el9)
> > and then just:
> > # grub2-mkconfig
> > and rebooting to see what's on the cmdline..?
> > 
> > The same files from a regular installation look like this (you'll notice
> > crashkernel is in both /etc/kernel/cmdline and /etc/default/grub while it's
> > missing from the AWS image even before the grub update):
> > 
> > [root@kvm-08-guest38 ~]# cat /proc/cmdline 
> > BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-319.el9.x86_64
> > root=/dev/mapper/rhel_kvm--08--guest38-root ro
> > crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
> > resume=/dev/mapper/rhel_kvm--08--guest38-swap
> > rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap
> > console=ttyS0,115200
> > 
> > [root@kvm-08-guest38 ~]# cat /etc/kernel/cmdline 
> > root=/dev/mapper/rhel_kvm--08--guest38-root ro
> > crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
> > resume=/dev/mapper/rhel_kvm--08--guest38-swap
> > rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap
> > console=ttyS0,115200 
> > 
> > [root@kvm-08-guest38 ~]# cat /etc/default/grub 
> > GRUB_TIMEOUT=5
> > GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
> > GRUB_DEFAULT=saved
> > GRUB_DISABLE_SUBMENU=true
> > GRUB_TERMINAL="serial console"
> > GRUB_SERIAL_COMMAND="serial --speed=115200"
> > GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
> > resume=/dev/mapper/rhel_kvm--08--guest38-swap
> > rd.lvm.lv=rhel_kvm-08-guest38/root rd.lvm.lv=rhel_kvm-08-guest38/swap
> > console=ttyS0,115200"
> > GRUB_DISABLE_RECOVERY="true"
> > GRUB_ENABLE_BLSCFG=true
> > 
> > The fix for 2162299 was to write the same contents to /etc/kernel/cmdline
> > and /etc/default/grub, but it looks like crashkernel got left out.
> 
> Thanks for the investigation.
> After run #grub2-mkconfig and reboot, the issue can be reproduced. 
> 
> Tomas, could you take a look from osbuild side?

Hi.

The behavior is not related to fixing bug #2162299, since osbuild-composer was never appending any "crashkernel" value to the kernel command line. We used to put there "crashkernel=auto", but this was removed due to bug #2006692.

AFAIK, the kexec-tools package is responsible for modifying the kernel command line and append the proper "crashkernel" value. So my guess is that the "92-crashkernel.install" and as a result "kdumpctl" is not handling the situation correctly, presumably due to changes in /etc/grub.d/10_linux which resulted in bug #2162299.

https://gitlab.com/redhat/centos-stream/rpms/kexec-tools/-/blob/c9s/kdumpctl#L1742-1768 should handle the situation, but it looks like it is not updating /etc/default/grub on new kernel install. 

osbuild-composer can't append the crashkernel value to the kernel cmdline in /etc/default/grub or /etc/kernel/cmdline, since we need to set it before installing any packages into the image fs tree, but the actual crashkernel value comes from /lib/modules/$kernel_ver/crashkernel.default installed by kernel-core.

Moving to kexec-tools as I think it is the proper tool to handle updating the kernel command line on kernel updates. There is nothing to be done from osbuild-composer's side from my PoV.

From the discussion, I see two different issues
1) upgrading from grub2-tools-2.06-61 to grub2-tools-2.06-63 causes a loss of the crashkernel parameter (comment#2)
2) Running grub2-mkconfig causes a loss of the crashkernel parameter (comment#4)

Issue 1) seems to be unique to an aws instance as I can't reproduce issue 1) in rhel-guest-image-9.3-20230531.0.x86_64.qcow2 so there should be something special about aws. Does anyone have a clue about it?

Issue 2) is expected because the agreement between osbuild and kexec-tools is if osbuid don't set custom crashkernel value in /etc/kernel/cmdline, then it's kexec-tools' responsibility to set up the crashekernel value. This agreement is implemented in https://gitlab.com/redhat/centos-stream/rpms/kexec-tools/-/blob/c9s/kdumpctl#L1809-1814. The implementation never updates /etc/default/grub so it's expected grub2-mkconfig will cause a loss of the crashkernel parameter. Btw, /lib/modules/$kernel_ver/crashkernel.default only existed for a short time and now it's kexec-tools who maintain the default crashkernel.

Two further observations,

1. Although this issue can't be reproduced on rhel-guest-image-9.3*, it can be reproduced on rhel-ec2-9.3* when upgrading grub2-tools from 2.06-61.el9.x86_64 to 2.06-63.el9.x86_64 for the first time (if we downgrade to 2.06-61 and then upgrade to 2.06-63, we can't reproduce it)

2. If we write the default crashkernel value to /etc/default/grub, upgrading grub2-tools from 2.06-61.el9.x86_64 to 2.06-63.el9.x86_64 won't cause a loss of the crashkernel parameter.

Hi guys,

I try to catch up with you. Please correct me if I’m wrong.

For the case of upgrading from grub2-tools-2.06-61 to grub2-tools-2.06-63,
-1. grub2-mkconfig runs, which reads /etc/default/grub and /etc/kernel/cmdline, and updates
-2. /boot/grub2/grub.cfg, finally, all the modification will be propagated to
-3. /boot/loader/entries/*.conf.

Thanks,

Pingfan

Hello Nicolas,

Could you Provide some insight on comment#13 ?

Thanks,

Pingfan

hi Pingfan,
Are these AWS instances BIOS or UEFI? or does this happen on both?
thanks.

(In reply to Marta Lewandowska from comment #15)
> hi Pingfan,
> Are these AWS instances BIOS or UEFI? or does this happen on both?
> thanks.

Neither I have access to AWS, nor I can find the console log in the bug description. So I just pass on your question to Libing.

@libhe, could you answer the question?

Thanks

Hi

For any grub experts, please share your opinion kindly, thanks in advance.

First, let me pay some words on the scene. After introduction of arm64 64k kernel variant, its crashkernel reserved memory size is significant different from the 4k variant. As a result, on the kdump side, it has different crashkernel="" for 64k and 4k.

Since there may be two variants coexist in the system, it can not use a unique "crashkernel=". Kdump tackles this issue by providing a installation hook, which calls a shell to decide the correct "crashkernel=" for the target kernel.


So far, it can be determined that neither "/etc/kernel/cmdline" nor "/etc/grub/default" can meet the requirement. 
After discussion with Dave and Coiby, we think that this issue may be resolved on the kdump side. 


As for the hook, where can it be place? Is "/etc/grub.d" a proper place and how can we ensure the hook is called after grub2-mkconfig resets the kernel cmdline for all entries?

Thanks,

Pingfan

(In reply to Pingfan Liu from comment #16)
> (In reply to Marta Lewandowska from comment #15)
> > hi Pingfan,
> > Are these AWS instances BIOS or UEFI? or does this happen on both?
> > thanks.
> 
> Neither I have access to AWS, nor I can find the console log in the bug
> description. So I just pass on your question to Libing.
> 
> @libhe, could you answer the question?
Our current released RHEL AMIs are boot from bios in x86 instances and uefi in arm instances.
This bug only appears in x86 instances.
> 
> Thanks

I think I understand most of what's going on...
While the x86 AWS is BIOS, it's configured as a hybrid, which means the ESP is mounted at /boot/efi

grub2-common has a posttrans scriptlet, which:
- checks for a mountpoint at /boot/efi and looks for a grub.cfg there
- if it's not there, it runs grub2-mkconfig to generate it
- it then checks whether the grub.cfg in the ESP is the so-called stub / unified config
- if not, then it creates it

We also have an issue that if one runs grub2-mkconfig, then the cmdline in bls entries get overwritten by whatever is in GRUB_CMDLINE_LINUX in /etc/default/grub: https://bugzilla.redhat.com/show_bug.cgi?id=2203203
This behavior can be changed by passing --no-grubenv-update to grub2-mkconfig.

I think that the posttrans scriptlet is wiping out crashkernel because it's not present in /etc/default/grub and that's why this can be reproduced by running grub2-mkconfig but only happens the first time and not during further downgrades / upgrades: because then the grub.cfg is already present in the ESP.

The only thing I don't get is why installing the newer image (RHEL-9.3.0-20230604.26) produces this issue instantly, but on the older image, one has to upgrade or run grub2-mkconfig. Did something change with the way grub is being installed?


here's the posttrans scriptlet for the curious ;)
%posttrans common
set -eu

EFI_HOME=%{efi_esp_dir}
GRUB_HOME=/boot/%{name}
ESP_PATH=/boot/efi

if ! mountpoint -q ${ESP_PATH}; then
    exit 0 # no ESP mounted, nothing to do
fi

if test ! -f ${EFI_HOME}/grub.cfg; then
    # there's no config in ESP, create one
    grub2-mkconfig -o ${EFI_HOME}/grub.cfg
fi

if grep -q "configfile" ${EFI_HOME}/grub.cfg; then
    exit 0 # already unified, nothing to do
fi

# create a stub grub2 config in EFI
BOOT_UUID=$(%{name}-probe --target=fs_uuid ${GRUB_HOME})
GRUB_DIR=$(%{name}-mkrelpath ${GRUB_HOME})

cat << EOF > ${EFI_HOME}/grub.cfg.stb
search --no-floppy --fs-uuid --set=dev ${BOOT_UUID}
set prefix=(\$dev)${GRUB_DIR}
export \$prefix
configfile \$prefix/grub.cfg
EOF

Hello Marta,

Thanks for the deep insight.

(In reply to Marta Lewandowska from comment #19)
> I think I understand most of what's going on...
> While the x86 AWS is BIOS, it's configured as a hybrid, which means the ESP
> is mounted at /boot/efi
> 
> grub2-common has a posttrans scriptlet, which:
> - checks for a mountpoint at /boot/efi and looks for a grub.cfg there
> - if it's not there, it runs grub2-mkconfig to generate it
> - it then checks whether the grub.cfg in the ESP is the so-called stub /
> unified config
> - if not, then it creates it
> 
> We also have an issue that if one runs grub2-mkconfig, then the cmdline in
> bls entries get overwritten by whatever is in GRUB_CMDLINE_LINUX in
> /etc/default/grub: https://bugzilla.redhat.com/show_bug.cgi?id=2203203
> This behavior can be changed by passing --no-grubenv-update to
> grub2-mkconfig.
> 
> I think that the posttrans scriptlet is wiping out crashkernel because it's
> not present in /etc/default/grub and that's why this can be reproduced by

Unfortunately, this will be always the truth after the introduction of the aarch64 64k kernel variant. Since there could be two variant kernels coexistent in the system, kdump can not decide which "crashkernel=" is set to "/etc/default/grub"

> running grub2-mkconfig but only happens the first time and not during
> further downgrades / upgrades: because then the grub.cfg is already present
> in the ESP.
> 
> The only thing I don't get is why installing the newer image
> (RHEL-9.3.0-20230604.26) produces this issue instantly, but on the older
> image, one has to upgrade or run grub2-mkconfig. Did something change with
> the way grub is being installed?
> 
> 
> here's the posttrans scriptlet for the curious ;)

It is helpful to navigate me there. After briefly browsing the grub2.spec, my coarse understanding is that the scripts under grub.d are used by grub2-mkconfig to generate grub.cfg.

So it is not proper to place a kdump hook there. But could there be a place for the kdump hook called by the posttrans in  grub2.spec, which handles the kernel parameter "crashkernel=" for each boot entry?

Or a simple therapy, as you mentioned, to pass --no-grubenv-update to grub2-mkconfig


Thanks,

Pingfan

> %posttrans common
> set -eu
> 
> EFI_HOME=%{efi_esp_dir}
> GRUB_HOME=/boot/%{name}
> ESP_PATH=/boot/efi
> 
> if ! mountpoint -q ${ESP_PATH}; then
>     exit 0 # no ESP mounted, nothing to do
> fi
> 
> if test ! -f ${EFI_HOME}/grub.cfg; then
>     # there's no config in ESP, create one
>     grub2-mkconfig -o ${EFI_HOME}/grub.cfg
> fi
> 
> if grep -q "configfile" ${EFI_HOME}/grub.cfg; then
>     exit 0 # already unified, nothing to do
> fi
> 
> # create a stub grub2 config in EFI
> BOOT_UUID=$(%{name}-probe --target=fs_uuid ${GRUB_HOME})
> GRUB_DIR=$(%{name}-mkrelpath ${GRUB_HOME})
> 
> cat << EOF > ${EFI_HOME}/grub.cfg.stb
> search --no-floppy --fs-uuid --set=dev ${BOOT_UUID}
> set prefix=(\$dev)${GRUB_DIR}
> export \$prefix
> configfile \$prefix/grub.cfg
> EOF

Hi Pingfan,

(In reply to Pingfan Liu from comment #20) 
> Unfortunately, this will be always the truth after the introduction of the
> aarch64 64k kernel variant. Since there could be two variant kernels
> coexistent in the system, kdump can not decide which "crashkernel=" is set
> to "/etc/default/grub"

This grub update to 2.06-63 is actually a patch that could be helpful here. (https://bugzilla.redhat.com/show_bug.cgi?id=2184069) It brings back the use of DEFAULTKERNEL in /etc/sysconfig/kernel so that the user can set which kernel variants will be set to default after updates. grub was previously just ignoring this variable.
In the current incarnation, it can handle: 64k|auto|rt|uki, which should also work for you, I guess? One possibility could be to check what is set there to default and then write the corresponding value for crashkernel into /etc/default/grub

I assume that installing a different kernel variant (4k vs 64k) will write the correct cmdline arguments to BLS snippets..? So the only risk to the above is if the user sets a different kernel variant as default using grubby and then runs grub2-mkconfig and gets their BLS config clobbered... which can happen of course, but maybe isn't so common.

I would like to discuss all this with Nicolas and others too. ;)

> It is helpful to navigate me there. After briefly browsing the grub2.spec,
> my coarse understanding is that the scripts under grub.d are used by
> grub2-mkconfig to generate grub.cfg.

that's right.
 
> So it is not proper to place a kdump hook there. But could there be a place
> for the kdump hook called by the posttrans in  grub2.spec, which handles the
> kernel parameter "crashkernel=" for each boot entry?

As long as crashkernel is set correctly in BLS snippets, it should work. The only issue I can foresee right now is what is mentioned above.

(In reply to Marta Lewandowska from comment #21)
> Hi Pingfan,
> 
> (In reply to Pingfan Liu from comment #20) 
> > Unfortunately, this will be always the truth after the introduction of the
> > aarch64 64k kernel variant. Since there could be two variant kernels
> > coexistent in the system, kdump can not decide which "crashkernel=" is set
> > to "/etc/default/grub"
> 
> This grub update to 2.06-63 is actually a patch that could be helpful here.
> (Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2184069) It brings back
> the use of DEFAULTKERNEL in /etc/sysconfig/kernel so that the user can set
> which kernel variants will be set to default after updates. grub was

kdump can not utilize that, since both 64k and 4k kernel can coexist with different "crashkernel="

> previously just ignoring this variable.
> In the current incarnation, it can handle: 64k|auto|rt|uki, which should
> also work for you, I guess? One possibility could be to check what is set
> there to default and then write the corresponding value for crashkernel into
> /etc/default/grub
> 
> I assume that installing a different kernel variant (4k vs 64k) will write
> the correct cmdline arguments to BLS snippets..? So the only risk to the

Yes, kdump decides the value of "crashkernel=" based on the installing kernel's name.

> above is if the user sets a different kernel variant as default using grubby
> and then runs grub2-mkconfig and gets their BLS config clobbered... which
> can happen of course, but maybe isn't so common.
> 

Yes but a slight adjustment, since no "crashkernel=" can be observed by grub2 (otherwise it should be aware of kernel parameter for different variants), it will swipe out all entries.

Thanks,

Pingfan

> I would like to discuss all this with Nicolas and others too. ;)
> 
> > It is helpful to navigate me there. After briefly browsing the grub2.spec,
> > my coarse understanding is that the scripts under grub.d are used by
> > grub2-mkconfig to generate grub.cfg.
> 
> that's right.
>  
> > So it is not proper to place a kdump hook there. But could there be a place
> > for the kdump hook called by the posttrans in  grub2.spec, which handles the
> > kernel parameter "crashkernel=" for each boot entry?
> 
> As long as crashkernel is set correctly in BLS snippets, it should work. The
> only issue I can foresee right now is what is mentioned above.

Hi Pingfan,
we have started to discuss this in a larger group, and will get back to you with some ideas soon. :)

Created attachment 1974476 [details]
patch

Hi,

We have a potential solution, which we are still discussing, in the attached patch. We have to make sure that it works correctly on AWS as well.
The idea is that if GRUB_ENABLE_BLSCFG=true (or is missing) in /etc/default/grub, then grub2-mkconfig will no longer overwrite the cmdline in BLS snippets by default. So, as long as the cmdline (including crashkernel) is set correctly during kernel installation, it should stay that way. :)  Does this seem reasonable to you for now?

We may also want to further protect / backup the cmdline for different kernel variants, like that there would be a separate file that stores the cmdline for each kernel variant that is installed. We need to think more how to implement this effectively, so this part is not ready yet.

(In reply to Marta Lewandowska from comment #24)
> Created attachment 1974476 [details]
> patch
> 
> Hi,
> 
> We have a potential solution, which we are still discussing, in the attached
> patch. We have to make sure that it works correctly on AWS as well.
> The idea is that if GRUB_ENABLE_BLSCFG=true (or is missing) in
> /etc/default/grub, then grub2-mkconfig will no longer overwrite the cmdline
> in BLS snippets by default. So, as long as the cmdline (including
> crashkernel) is set correctly during kernel installation, it should stay
> that way. :)  Does this seem reasonable to you for now?
> 

It can satisfy kdump's requirement.

> We may also want to further protect / backup the cmdline for different
> kernel variants, like that there would be a separate file that stores the
> cmdline for each kernel variant that is installed. We need to think more how
> to implement this effectively, so this part is not ready yet.

Probably, the RO attribute of some directories such as /lib/modules/$kernel-version is a obstacle to this attempt.


At least, we find a solution to this bug. Thank you and your team for the quick response.


Pingfan

Based on comment#24, move it to the grub component

There is a new reported bug: https://bugzilla.redhat.com/show_bug.cgi?id=2221543

I think it shares the same root cause as this one

Hi Bootloader team/Marta,

Do you have detail plan on the fix (better within rhel9.3)? 
Another kdump CUT bug(bz2221543) also depends on the candidate solution.

Thanks.

(In reply to Marta Lewandowska from comment #24)
> Created attachment 1974476 [details]
> patch
> 
> Hi,
> 
> We have a potential solution, which we are still discussing, in the attached
> patch. We have to make sure that it works correctly on AWS as well.
> The idea is that if GRUB_ENABLE_BLSCFG=true (or is missing) in
> /etc/default/grub, then grub2-mkconfig will no longer overwrite the cmdline
> in BLS snippets by default. So, as long as the cmdline (including
> crashkernel) is set correctly during kernel installation, it should stay
> that way. :)  Does this seem reasonable to you for now?
> 
> We may also want to further protect / backup the cmdline for different
> kernel variants, like that there would be a separate file that stores the
> cmdline for each kernel variant that is installed. We need to think more how
> to implement this effectively, so this part is not ready yet.

(In reply to Pingfan Liu from comment #13)
Hi Pingfan,
Sorry for the delay.

> Hi guys,
> 
> I try to catch up with you. Please correct me if I’m wrong.
> 
> For the case of upgrading from grub2-tools-2.06-61 to grub2-tools-2.06-63,
> -1. grub2-mkconfig runs, which reads /etc/default/grub and
> /etc/kernel/cmdline, and updates

I am not sure that grub2-mkconfig itself reads /etc/kernel/cmdline.
Additionally to using /etc/default/grub, it's also using templates
located in /etc/grub.d/

> -2. /boot/grub2/grub.cfg, finally, all the modification will be propagated to
Correct
> -3. /boot/loader/entries/*.conf.
Correct
> 
> Thanks,
> 
> Pingfan

Nicolas

(In reply to Jie Li from comment #28)
> Hi Bootloader team/Marta,
> 
> Do you have detail plan on the fix (better within rhel9.3)? 
> Another kdump CUT bug(Red Hatbz2221543) also depends on the candidate
> solution.
> 
> Thanks.
> 

Hi Jie, 
We don't have a plan yet, unfortunately.
We're figuring out how to handle it in a timely manner and don't break other stuff.

Petr

Hello Libing,

As you look being the one having access to AWS instances,
are you able to build your own image to test Marta's patch from comment 24 ?

(In reply to Petr Janda from comment #31)
> Hello Libing,
> 
> As you look being the one having access to AWS instances,
> are you able to build your own image to test Marta's patch from comment 24 ?

Hi Petr,

We can't build own image to test the patch. What we can do is to find a reproducible image, and then install the grub2 with the patch to verify if no param is lost after installation, and the kernel is upgraded, it will not be lost.

Is it okay to verify this patch in this way?

I believe it is OK to test it this way.

Thank you very much for testing! I'm glad it works.

I want to see whether this works as expected for another bug (bz#2221543), and then ask for an official build that we can do further testing on.

Created attachment 1976266 [details]
patch v2

fix patch so that --update-bls-cmdline argument actually works and is mentioned in 'usage'

Reproduced with grub2-2.06-65.el9 in RHEL-9.3.0-20230725.27

# grubby --update-kernel /boot/vmlinuz-5.14.0-344.el9.ppc64le --args "args4u"

# cat /etc/default/grub 
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="ofconsole"
GRUB_CMDLINE_LINUX="crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINFO="terminfo -g 80x24 console"
GRUB_DISABLE_OS_PROBER=true

# grub2-mkconfig -o /etc/grub2.cfg
# grubby --info /boot/vmlinuz-5.14.0-344.el9.ppc64le 
index=0
kernel="/boot/vmlinuz-5.14.0-344.el9.ppc64le"
args="ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub"
root="/dev/mapper/rhel_ibm--p9z--26--lp12-root"
initrd="/boot/initramfs-5.14.0-344.el9.ppc64le.img"
title="Red Hat Enterprise Linux (5.14.0-344.el9.ppc64le) 9.3 (Plow)"
id="89f7902ad50a42f89ad85141cfaa3612-5.14.0-344.el9.ppc64le"

(BLS cmdline gets overwritten)

rebooting

# cat /proc/cmdline 
BOOT_IMAGE=(ieee1275//vdevice/v-scsi@30000003/disk@8100000000000000,msdos2)/vmlinuz-5.14.0-344.el9.ppc64le root=/dev/mapper/rhel_ibm--p9z--26--lp12-root ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub


Tested using grub2-2.06-67.el9 using the same steps and BLS snippets are not overwritten.

rebooting:
# cat /proc/cmdline 
BOOT_IMAGE=(ieee1275//vdevice/v-scsi@30000003/disk@8100000000000000,msdos2)/vmlinuz-5.14.0-344.el9.ppc64le root=/dev/mapper/rhel_ibm--p9z--26--lp12-root ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap args4u

# grub2-mkconfig -o /etc/grub2.cfg --update-bls-cmdline
# grubby --info /boot/vmlinuz-5.14.0-344.el9.ppc64le 
index=0
kernel="/boot/vmlinuz-5.14.0-344.el9.ppc64le"
args="ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub"
root="/dev/mapper/rhel_ibm--p9z--26--lp12-root"
initrd="/boot/initramfs-5.14.0-344.el9.ppc64le.img"
title="Red Hat Enterprise Linux (5.14.0-344.el9.ppc64le) 9.3 (Plow)"
id="89f7902ad50a42f89ad85141cfaa3612-5.14.0-344.el9.ppc64le"

If GRUB_ENABLE_BLSCFG=false then BLS snippets are also not overwritten, but rebooting gets args from /etc/default/grub to kernel:
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-5.14.0-344.el9.ppc64le root=/dev/mapper/rhel_ibm--p9z--26--lp12-root ro crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_ibm-p9z-26-lp12/root rd.lvm.lv=rhel_ibm-p9z-26-lp12/swap etc_def_grub

Tested on both ppc64le and x86_64

Created attachment 1982321 [details]
patch v3

Third time's the charm

Aware of it, still working on fix.

Reproduced with grub2-2.06-68.el9: grub2-mkconfig updates BLS cmdline by default 

[root@sweetpig-9 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial"
GRUB_CMDLINE_LINUX="console=tty0 elevator=noop crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_sweetpig--9-swap rd.lvm.lv=rhel_sweetpig-9/root rd.lvm.lv=rhel_sweetpig-9/swap console=ttyS0 etc_def_grub"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@sweetpig-9 ~]# grubby --update-kernel /boot/vmlinuz-5.14.0-350.el9.x86_64 --args "some_args"
[root@sweetpig-9 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro console=tty0 elevator=noop crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_sweetpig--9-swap rd.lvm.lv=rhel_sweetpig-9/root rd.lvm.lv=rhel_sweetpig-9/swap console=ttyS0 some_args"
root="/dev/mapper/rhel_sweetpig--9-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="9cc32a47feed477e95d52efa8e78b0b0-5.14.0-350.el9.x86_64"

[root@sweetpig-9 ~]# grub2-mkconfig -o /etc/grub2.cfg 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

[root@sweetpig-9 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro console=tty0 elevator=noop crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_sweetpig--9-swap rd.lvm.lv=rhel_sweetpig-9/root rd.lvm.lv=rhel_sweetpig-9/swap console=ttyS0 etc_def_grub"
root="/dev/mapper/rhel_sweetpig--9-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="9cc32a47feed477e95d52efa8e78b0b0-5.14.0-350.el9.x86_64"


Verified (on x86, ppc64le, aarch64) with grub2-2.06-70.el9_3: BLS cmdline only updated (correctly) in the installer environment and upon user request -- not by default
[root@kvm-04-guest21 ~]# grep CMDLINE /etc/default/grub 
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200"

[root@kvm-04-guest21 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-350.el9.x86_64 root=/dev/mapper/rhel_kvm--04--guest21-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200

[root@kvm-04-guest21 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_kvm--04--guest21-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 

[root@kvm-04-guest21 ~]# grubby --update-kernel /boot/vmlinuz-5.14.0-350.el9.x86_64 --args "some_args"
[root@kvm-04-guest21 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 some_args"
root="/dev/mapper/rhel_kvm--04--guest21-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="65d1391391aa4d62bc35c50923807afb-5.14.0-350.el9.x86_64"
 
[root@kvm-04-guest21 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 etc_def_grub"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@kvm-04-guest21 ~]# grub2-mkconfig -o /etc/grub2.cfg 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

[root@kvm-04-guest21 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 some_args"
root="/dev/mapper/rhel_kvm--04--guest21-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="65d1391391aa4d62bc35c50923807afb-5.14.0-350.el9.x86_64"

[root@kvm-04-guest21 ~]# grub2-mkconfig -o /etc/grub2.cfg --update-bls-cmdline
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

[root@kvm-04-guest21 ~]# grubby --info /boot/vmlinuz-5.14.0-350.el9.x86_64 
index=0
kernel="/boot/vmlinuz-5.14.0-350.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--04--guest21-swap rd.lvm.lv=rhel_kvm-04-guest21/root rd.lvm.lv=rhel_kvm-04-guest21/swap console=ttyS0,115200 etc_def_grub"
root="/dev/mapper/rhel_kvm--04--guest21-root"
initrd="/boot/initramfs-5.14.0-350.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-350.el9.x86_64) 9.3 (Plow)"
id="65d1391391aa4d62bc35c50923807afb-5.14.0-350.el9.x86_64"

Re-built with correct tag, but same content

$ diff -q grub-9_3 grub-9_3.1
Only in grub-9_3.1: grub2-2.06-70.el9_3.1.src.rpm
Only in grub-9_3: grub2-2.06-70.el9_3.src.rpm
Files grub-9_3/grub2.spec and grub-9_3.1/grub2.spec differ

$ diff grub-9_3/grub2.spec grub-9_3.1/grub2.spec
19c19
< Release:	70%{?dist}
---
> Release:	70%{?dist}.1
535a536,541
> * Thu Sep 7 2023 Nicolas Frayer <nfrayer> - 2.06-70.el9_3.1
> - Bump spec release version
> - Related: #2203203
> - Related: #2212320
> - Related: #2221543
> 

gating tests grub2 / etc_default_grub / kernel_cmdline passed, so it is working as expected.

[root@kvm-07-guest39 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--07--guest39-swap rd.lvm.lv=rhel_kvm-07-guest39/root rd.lvm.lv=rhel_kvm-07-guest39/swap console=ttyS0,115200"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@kvm-07-guest39 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-362.el9.x86_64 root=/dev/mapper/rhel_kvm--07--guest39-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_kvm--07--guest39-swap rd.lvm.lv=rhel_kvm-07-guest39/root rd.lvm.lv=rhel_kvm-07-guest39/swap console=ttyS0,115200

grub2-2.06-70.el9_3.1 is in RHEL-9.3.0-20230908.62 nightly, on which we ran our Tier2 automated + manual tests
Moving to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (grub2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6653