Bug 2006692

Summary: Please remove deprecated option "crashkernel=auto" from rhel9 distros in image builder
Product: Red Hat Enterprise Linux 9 Reporter: Frank Liang <xiliang>
Component: osbuild-composerAssignee: Tomáš Hozza <thozza>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.0CC: coxu, jrusz, linl, obudai, qzhang, ruyang, thozza, vkuznets, wshi, ymao, yuxisun
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: osbuild-composer-44-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 13:29:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2024976    
Bug Blocks:    

Description Frank Liang 2021-09-22 08:07:14 UTC
Description of problem:
From bz1942398, this option was dropped from rhel9, so please remove it when generating rhel9 images.
[root@dev209 test]# guestfish -a rhel-guest-image-9.0-20210921.6.x86_64.qcow2 -i sh 'cat /boot/grub2/grubenv'
# GRUB Environment Block
kernelopts=root=UUID=95328833-60be-4de1-af21-8ac2d7dc6a37 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto
saved_entry=ffffffffffffffffffffffffffffffff-5.14.0-1.2.1.el9.x86_64

RHEL Version:
RHEL-9.0(5.14.0-1.2.1.el9)

How reproducible:
100%

Steps to Reproduce:
1. Check grubenv cfg in rhel9 images

Actual results:
crashkernel=auto applied

Expected results:
no crashkernel=auto required any more

Additional info:
- N/A

Comment 1 Tomáš Hozza 2021-09-22 09:25:09 UTC
Created upstream issue https://github.com/osbuild/osbuild-composer/issues/1819

Comment 2 Frank Liang 2021-09-26 09:13:39 UTC
Hi Tomas,

To keep consistent with the previous images with kdump enabled, I guess you need to specify crashkernel after removed 'crashkernel=auto'.

I am asking kairui for more suggestions.

Hi Kairui,

Do you have any comments in specifying it? Because I found it was not always added when new kernel installed.

eg."crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M" on x86_64, how about other platforms?

Comment 3 Kairui Song 2021-10-26 10:02:41 UTC
Hi,

Starting from RHEL9, crashkernel will have a default value included in the kernel package, on a installed system there will be a file available as /lib/modules/$(uname -r)/crashkernel.default which contains the default crashkernel value for that specified kernel version. System installer should set crashkernel value based on this file.

You can check bz1974638 and bz1942398 for more info.

Comment 5 Tomáš Hozza 2021-11-02 11:59:12 UTC
Hi Frank.

Please correct me if I'm wrong. After going through the bugs linked in comment #3, it seems to me that Image builder should simply append the content of "${rootfs}/lib/modules/${kernelver}/crashkernel.default" provided by the installed kernel package to the kernel command line options used by the image. Is that correct?

Comment 6 Frank Liang 2021-11-02 12:28:03 UTC
(In reply to Tomáš Hozza from comment #5)
> Hi Frank.
> 
> Please correct me if I'm wrong. After going through the bugs linked in
> comment #3, it seems to me that Image builder should simply append the
> content of "${rootfs}/lib/modules/${kernelver}/crashkernel.default" provided
> by the installed kernel package to the kernel command line options used by
> the image. Is that correct?
Yes, I think so.

Here are examples of my systems:
# cat /lib/modules/5.14.0-8.el9.x86_64/crashkernel.default 
crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
# cat /lib/modules/5.14.0-11.el9.aarch64/crashkernel.default 
crashkernel=2G-:448M

Comment 7 Tomáš Hozza 2021-11-12 14:08:05 UTC
There have been multiple issued identified with this change, which will require changes to osbuild as well as image definitions in osbuild-composer. More information is in the upstream issue https://github.com/osbuild/osbuild-composer/issues/1819#issuecomment-966199997

Comment 9 Dave Young 2022-01-13 05:44:35 UTC
There is a new change in RHEL9.0GA in bug https://bugzilla.redhat.com/show_bug.cgi?id=1895258, the /lib/modules/.../crashkernel.default has been dropped as well,  all have been moved to kexec-tools. There is a kdumpctl option to get the correct default values.

Coiby, can you add more details here about how to get the proper value for os-builder?

Comment 10 Coiby 2022-01-14 01:51:58 UTC
Hi Tomáš,

kexec-tools now would set up kernel crashkernel for osbuild in kernel hook [1] as resolved in Bug 2024976. So now osbuild can simply drop crashkernel=auto and there is no need for osbuild to retrieve the default crashkernel value itself.
    
[1] https://src.fedoraproject.org/rpms/kexec-tools/c/ddd428a1d0d72bee2a8459b1a81541bcd0676873?branch=rawhide

Comment 11 Tomáš Hozza 2022-02-04 16:10:15 UTC
In the interest of time I went ahead and submitted a change in upstream - https://github.com/osbuild/osbuild-composer/pull/2262

Once Bug #2024976 is resolved, the appropriate crashkernel value will be set when RHEL-9.0 image is built.

Comment 12 Tomáš Hozza 2022-02-04 16:13:13 UTC
Note for QE:

The step to verify this change are:
- Before Bug #2024976 is resolved, check that RHEL EC2 and guest image do not have 'crashkernel=auto' on the kernel command line. In fact there won't be any 'crashkernel' option set.
- After Bug #2024976 is resolved, check that RHEL EC2 and guest image do not have 'crashkernel=auto' on the kernel command line. The 'crashkernel' option will be set to the appropriate value by kexec-tools package.

Comment 13 Jakub Rusz 2022-02-04 16:43:17 UTC
This is covered by image-tests.sh so fairly easy to verify for QE.

Comment 14 Frank Liang 2022-03-01 07:46:42 UTC
Uploaded the latest RHEL-9.0-20220227.3(rhel-ec2-9.0-20220227.3.x86_64.raw) to ec2 and crashkernel is set correctly.
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-69.el9.x86_64 root=UUID=da88f4cb-fcdd-4f87-9eb7-40155e2c75bc console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

Comment 16 Ondřej Budai 2022-03-01 12:23:30 UTC
Fixed by osbuild-composer 44.

Comment 23 errata-xmlrpc 2022-05-17 13:29:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: osbuild-composer), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2522