Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2233095

Summary:	After 16.2 update RHEL booting into wrong kernel version on all compute nodes
Product:	Red Hat OpenStack	Reporter:	Joaquín Veira <jveiraca>
Component:	openstack-tripleo-heat-templates	Assignee:	Mikolaj Ciecierski <mciecier>
Status:	CLOSED ERRATA	QA Contact:	Archana Singh <arcsingh>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.2 (Train)	CC:	arcsingh, bshephar, jpretori, kgilliga, mariel, mburns, mciecier, pgrist, prgutier, sgolovat, tarcher, tvignaud
Target Milestone:	z6	Keywords:	Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.6.1-2.20230808225216.el8ost	Doc Type:	Bug Fix
Doc Text:	On UEFI-based systems, Compute nodes were not booting with the latest kernel version because `/boot/grub2/grubenv` is not a symlink to `/boot/efi/EFI/redhat/grubenv`. With this update, the symlink from `/boot/grub2/grubenv` to `/boot/efi/EFI/redhat/grubenv` is recreated during an update. As a result, the version of the kernel in the `saved_entry` field of `/boot/efi/EFI/redhat/grubenv` is updated, and the Compute node boots with the correct kernel version.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-11-08 19:19:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joaquín Veira 2023-08-21 12:28:58 UTC

Description of problem:
- During boot compute node is not booting with latest kernel: 4.18.0-305.76.1.el8_4.x86_64 having index:0, but booting with older kernel: 4.18.0-305.49.1.el8_4.x86_64 with index-1
- grubby configuration shows that the kernel booted should be 4.18.0-305.76.1.el8_4.x86_64
- This is UEFI based setup.
- The latest kernel is selected in two conditions.
A.) When latest version is selected manually during grub prompt.
B.) In the  grubenv.rpmnew -> ../efi/EFI/redhat/grubenv[1] which has saved_entry has  saved_entry=9a297d8fa6df92258e47f64024c09d0e-4.18.0-305.49.1.el8_4.x86_64. If this entry is  edited and removed rebooting the node gets to the latest kernel. 

Customer fixed it by checking and following: https://access.redhat.com/solutions/4326431

# grubby --default-kernel
/boot/vmlinuz-4.18.0-305.76.1.el8_4.x86_64

#  grubby --info=ALL | grep title
title="Red Hat Enterprise Linux (4.18.0-305.76.1.el8_4.x86_64) 8.4 (Ootpa)" <<<<<<<<<<<<<<<<Index 0
title="Red Hat Enterprise Linux (4.18.0-305.49.1.el8_4.x86_64) 8.4 (Ootpa)"
title="Red Hat Enterprise Linux (4.18.0-193.64.1.el8_2.x86_64) 8.2 (Ootpa)"
title="Red Hat Enterprise Linux (0-rescue-9a297d8fa6df92258e47f64024c09d0e) 8.2 (Ootpa)"
title="Red Hat Enterprise Linux (4.18.0-193.51.1.el8_2.x86_64) 8.2 (Ootpa)"
title="Red Hat Enterprise Linux (0-rescue-0a3a9377748d4006be2bf2764fc6d219) 8.2 (Ootpa)"

# grubby --default-index
0

**But, uname  shows Index-1**
# uname -a
Linux hostname.xxx.xxx 4.18.0-305.49.1.el8_4.x86_64 #1 SMP Wed May 11 09:47:48 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
#

**The kernel version is changed in grubenv also**
$ cat /boot/grub2/grubenv
# GRUB Environment Block
kernelopts=root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ro console=ttyS0 console=ttyS0,115200n81 no_timer_check  crashkernel=auto rhgb quiet  tsx=off default_hugepagesz=1G hugepagesz=1G hugepages=384 intel_iommu=on iommu=pt   audit=1 audit_backlog_limit=8192 random.trust_cpu=on pti=on page_poison=1 slub_debug=P
boot_indeterminate=1
boot_success=1
saved_entry=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-4.18.0-305.76.1.el8_4.x86_64 <<<<<<<<<<<<<<<<<<

Version-Release number of selected component (if applicable):
RHOSP 16.2

How reproducible:
Minor update from 16.1 to 16.2

Actual results:
Wrong boot entry in grub

Expected results:
Should boot with latest kernel

Additional info:
Sosreports and /var/lib/mistral attached to case

Comment 19 Thierry Vignaud 2023-09-27 13:57:08 UTC

Once it got reviewed, we can merge it.

Comment 34 errata-xmlrpc 2023-11-08 19:19:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.2.6 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6307