Bug 2046456

Summary:	[OSP16.2] UEFI Compute nodes booting RHEL 8.2 Kernel instead of new 8.4 one
Product:	Red Hat OpenStack	Reporter:	ggrimaux
Component:	openstack-tripleo-heat-templates	Assignee:	Lukas Bezdicka <lbezdick>
Status:	ON_QA ---	QA Contact:	Khomesh Thakre <kthakre>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	16.2 (Train)	CC:	alisci, bwelterl, jlarriba, jpretori, mburns, svigan
Target Milestone:	z6	Keywords:	Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.6.1-2.20230504005012.8e38b2e.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description ggrimaux 2022-01-26 17:55:23 UTC

Description of problem:
Client on OSP13 with UEFI nodes used grubby on its compute nodes to modify an option.
Client did a FFU 13 to 16.1 last fall.
Yesterday they did a minor update from 16.1.5 to 16.2.1.

On reboot of those compute nodes they would boot the RHEL8.2 kernel even though 'grubby --default-kernel' would show 8.4 being the default one.

To fix this we did the following:
rm /boot/efi/EFI/redhat/grubenv
mv /boot/grub2/grubenv /boot/efi/EFI/redhat/grubenv
ln -s /boot/grub2/grubenv /boot/efi/EFI/redhat/grubenv

And reboot would now default to RHEL 8.4 kernel.

A patach has been written here and might be fixing this situation:
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/826483

We have sosreport of compute nodes having this issue.

Version-Release number of selected component (if applicable):
OSP16.2.1

How reproducible:
100%

Steps to Reproduce:
1. OSP13 + grubby to modify the config
2. FFU to 16.1
3. Minor to 16.2
4. Reboot node.

Actual results:
Node would boot RHEL 8.2 kernel instead of new RHEL 8.4 kernel.

Expected results:
Boot the new RHEL 8.4 kernel.

Additional info:
sosreport are attached to another case. Will put that information in next private comment.

Comment 5 Lukas Bezdicka 2023-03-07 15:09:34 UTC

I ended up circling back to this due to 16->17 upgrade.

The copy of grubenv instead of linking was due to some grub bug which probably all envs no longer hit so we should patch update and upgrade in 16.2.