RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2227722 - kickstart installation produces an invalid /boot/loader/entries/XXX.conf file
Summary: kickstart installation produces an invalid /boot/loader/entries/XXX.conf file
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: grub2
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Bootloader engineering team
QA Contact: Release Test Team
URL:
Whiteboard:
: 2228847 2229122 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-31 08:16 UTC by Lars
Modified: 2023-08-26 11:42 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-09 07:19:41 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
kickstart file (1.84 KB, text/x-matlab)
2023-07-31 08:16 UTC, Lars
no flags Details
package diff (30.47 KB, text/plain)
2023-07-31 09:23 UTC, Lars
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-163864 0 None None None 2023-07-31 08:17:47 UTC

Description Lars 2023-07-31 08:16:09 UTC
Created attachment 1980832 [details]
kickstart file

Description of problem:

Running an kickstart installation produces an invalid /boot/loader/entries/XXX.conf file which causes boot problems.

The options line contains the option from the kickstart and not for the final OS.
e.g.:
options inst.ks=http://192.168.0.4/autoinst/testcentos9.cfg inst.text net.ifnames=0 biosdevname=0 selinux=0 nameserver=192.168.0.4 ip=10.70.56.13::10.70.56.1:255.255.255.224:testcentos9:eth0:none



Version-Release number of selected component (if applicable):
Compose_ID CentOS-Stream-9-20230727.1

How reproducible:
Run a kickstart installation with the attached ks.cfg file.
We are booting from the ISO image inside a VMware VM.

Steps to Reproduce:
1. Modify the ks.cfg to you test environment and place the file on webserver
2. Boot up from the ISO
3. Use the following options
inst.ks=http://192.168.0.4/autoinst/testcentos9.cfg inst.text net.ifnames=0 biosdevname=0 selinux=0 nameserver=192.168.0.4 ip=10.70.56.13::10.70.56.1:255.255.255.224:testcentos9:eth0:none


Actual results:
anaconda installs with an non functional /boot/loader/entries/XXX.conf
System will not boot up (e.g. initrd-switch-root.service fails)


Expected results:
anaconda installs the system with functional /boot/loader/entries/XXX.conf, system will boot


Additional info:

Comment 1 farrotin 2023-07-31 09:04:16 UTC
the CentOS Stream 9 test environment also confirm that it's not installable : https://testing.stream.centos.org/ : all failing for cs9 deployments.
Manually redeployed one of the kvm guests and can confirm I see the same problem reported in this ticket

Comment 2 Lars 2023-07-31 09:23:37 UTC
Created attachment 1980836 [details]
package diff

Comment 3 Lars 2023-07-31 09:26:08 UTC
I did some tests with local freezed BaseOS copy from 20230706.
Installing with the latest install.img against the local freezed copy is working fine.


I attached a list of new packages between my old copy and the current BaseOS.
Hopefully this helps to find the root cause.

Comment 4 Adam Samalik 2023-07-31 12:01:22 UTC
Quickly looking at older runs on testing.stream.centos.org, the compose CentOS-Stream-9-20230720.0 installed fine (even though the tests failed later on): https://testing.stream.centos.org/job/CentOS_9-stream_x86_64_kvm/22/consoleFull

Comment 5 Adam Samalik 2023-07-31 12:24:46 UTC
Also the DVD (CentOS-Stream-9-20230727.1-aarch64-dvd1.iso) installs fine using Anaconda, I tested that on Friday.

Comment 6 Jan Stodola 2023-07-31 12:53:00 UTC
Reproduced by running a manual installation from https://odcs.stream.centos.org/production/CentOS-Stream-9-20230731.0/compose/BaseOS/x86_64/os/ in a VM (KVM), using the defaults in the installer.

The kernel boot arguments on the installed system are the ones used for the installation: "inst.repo=https://odcs.stream.centos.org/.../x86_64/os/".

Reassigning to grub2.

Comment 8 Huijuan Zhao 2023-08-02 01:51:28 UTC
Also meet this issue when deploy instance with CentOS-Stream-GenericCloud-9-20230727.1.x86_64.qcow2 on OpenStack PSI:

Failing to boot the instance, it enters to dracut and reports "can't mount root filesystem"

Comment 9 Robby Callicotte 2023-08-02 12:47:49 UTC
I was performing a test install of CentOS Stream 9 using the latest[1] 
available dvd iso from the mirror[2].

I manually select "minimal install" from the selectable packages.  The install 
seems to go smoothly.  Upon reboot I am greeted with:

[FAILED] Failed to start Switch Root

Further examination of the switch-root service shows the following:

Failed to switch root: Specified switch root path '/sysroot' does not seem to 
be an OS tree.  os-release file is missing.

Booting into rescue media shows that the os-release file(s) actually do exist:

ls -l /etc/os-release
lwrxwrxwrx.  1 root root 21 Apr   6 08:27 /etc/os-release  -> ../usr/lib/os-
release

ls -l /usr/lib/os-release
-rw-r--r--.  1 root root 391 Apr   6 08:27 /usr/lib/os-release


[1] - https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/iso/CentOS-Stream-9-latest-x86_64-dvd1.iso
[2] - https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/iso/

Comment 10 Adam Samalik 2023-08-03 07:50:11 UTC
Tagged the older version of grub2 temporarily into c9s and running a new c9s production compose now

Comment 11 Jan Stodola 2023-08-04 09:58:42 UTC
*** Bug 2229122 has been marked as a duplicate of this bug. ***

Comment 12 Troy Dawson 2023-08-07 18:16:57 UTC
I have verified the failure.
I have finished a rebuild of the CentOS Stream 9 compose that has the latest grub in it.
I have verified that this latest compose (CentOS-Stream-9-20230807.1) installs and boots correctly, using the same procedure that failed before.
In short, I believe this latest compose has this fixed.

I have already pushed this compose to the repo mirrors.
We have an 8 hour delay so that the repo's can propagate around the world before we push out the images to our various places.
If you want to test the various images before the 8 hours they are here
https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/

Comment 13 Martin Jackson 2023-08-07 20:44:15 UTC
I have confirmed the fix with newly synced repos (my installation was via Kickstart as in the original report).  System now boots correctly after kickstart installation. Thanks Troy!

Comment 14 Lars 2023-08-08 06:06:02 UTC
I can also confirm that the issue is solved with the latest compose (20230807.1). Thanks!

Comment 15 Troy Dawson 2023-08-08 21:11:26 UTC
This issue is now fixed, both in the CentOS Stream 9 repository, and in all the CentOS Stream 9 images.
I'm not sure if I should close this, or if the Bootloader team needs to do something before it's closed.

Comment 16 Marta Lewandowska 2023-08-09 07:19:41 UTC
Thanks to everyone who caught and, importantly, fixed this! (New version of the patch that caused this has been tested more thoroughly, and will not go into a compose before further testing is performed)

I am closing this one now... sorry for the hassle.

Comment 17 Marta Lewandowska 2023-08-09 07:21:03 UTC
*** Bug 2228847 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.