Bug 2233095 - After 16.2 update RHEL booting into wrong kernel version on all compute nodes
Summary: After 16.2 update RHEL booting into wrong kernel version on all compute nodes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z6
: 16.2 (Train on RHEL 8.4)
Assignee: Mikolaj Ciecierski
QA Contact: Archana Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-21 12:28 UTC by Joaquín Veira
Modified: 2023-11-08 19:19 UTC (History)
12 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.6.1-2.20230808225216.el8ost
Doc Type: Bug Fix
Doc Text:
On UEFI-based systems, Compute nodes were not booting with the latest kernel version because `/boot/grub2/grubenv` is not a symlink to `/boot/efi/EFI/redhat/grubenv`. With this update, the symlink from `/boot/grub2/grubenv` to `/boot/efi/EFI/redhat/grubenv` is recreated during an update. As a result, the version of the kernel in the `saved_entry` field of `/boot/efi/EFI/redhat/grubenv` is updated, and the Compute node boots with the correct kernel version.
Clone Of:
Environment:
Last Closed: 2023-11-08 19:19:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 895847 0 None MERGED [train-only] Fix grubenv linking for minor update 2023-09-25 08:50:11 UTC
Red Hat Issue Tracker OSP-27592 0 None None None 2023-08-21 12:37:32 UTC
Red Hat Knowledge Base (Solution) 7034430 0 None None None 2023-10-08 07:18:14 UTC
Red Hat Product Errata RHBA-2023:6307 0 None None None 2023-11-08 19:19:37 UTC

Description Joaquín Veira 2023-08-21 12:28:58 UTC
Description of problem:
- During boot compute node is not booting with latest kernel: 4.18.0-305.76.1.el8_4.x86_64 having index:0, but booting with older kernel: 4.18.0-305.49.1.el8_4.x86_64 with index-1
- grubby configuration shows that the kernel booted should be 4.18.0-305.76.1.el8_4.x86_64
- This is UEFI based setup.
- The latest kernel is selected in two conditions.
A.) When latest version is selected manually during grub prompt.
B.) In the  grubenv.rpmnew -> ../efi/EFI/redhat/grubenv[1] which has saved_entry has  saved_entry=9a297d8fa6df92258e47f64024c09d0e-4.18.0-305.49.1.el8_4.x86_64. If this entry is  edited and removed rebooting the node gets to the latest kernel. 

Customer fixed it by checking and following: https://access.redhat.com/solutions/4326431

# grubby --default-kernel
/boot/vmlinuz-4.18.0-305.76.1.el8_4.x86_64

#  grubby --info=ALL | grep title
title="Red Hat Enterprise Linux (4.18.0-305.76.1.el8_4.x86_64) 8.4 (Ootpa)" <<<<<<<<<<<<<<<<Index 0
title="Red Hat Enterprise Linux (4.18.0-305.49.1.el8_4.x86_64) 8.4 (Ootpa)"
title="Red Hat Enterprise Linux (4.18.0-193.64.1.el8_2.x86_64) 8.2 (Ootpa)"
title="Red Hat Enterprise Linux (0-rescue-9a297d8fa6df92258e47f64024c09d0e) 8.2 (Ootpa)"
title="Red Hat Enterprise Linux (4.18.0-193.51.1.el8_2.x86_64) 8.2 (Ootpa)"
title="Red Hat Enterprise Linux (0-rescue-0a3a9377748d4006be2bf2764fc6d219) 8.2 (Ootpa)"

# grubby --default-index
0

**But, uname  shows Index-1**
# uname -a
Linux hostname.xxx.xxx 4.18.0-305.49.1.el8_4.x86_64 #1 SMP Wed May 11 09:47:48 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
#

**The kernel version is changed in grubenv also**
$ cat /boot/grub2/grubenv
# GRUB Environment Block
kernelopts=root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ro console=ttyS0 console=ttyS0,115200n81 no_timer_check  crashkernel=auto rhgb quiet  tsx=off default_hugepagesz=1G hugepagesz=1G hugepages=384 intel_iommu=on iommu=pt   audit=1 audit_backlog_limit=8192 random.trust_cpu=on pti=on page_poison=1 slub_debug=P
boot_indeterminate=1
boot_success=1
saved_entry=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-4.18.0-305.76.1.el8_4.x86_64 <<<<<<<<<<<<<<<<<<

Version-Release number of selected component (if applicable):
RHOSP 16.2

How reproducible:
Minor update from 16.1 to 16.2

Actual results:
Wrong boot entry in grub

Expected results:
Should boot with latest kernel

Additional info:
Sosreports and /var/lib/mistral attached to case

Comment 19 Thierry Vignaud 2023-09-27 13:57:08 UTC
Once it got reviewed, we can merge it.

Comment 34 errata-xmlrpc 2023-11-08 19:19:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.2.6 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6307


Note You need to log in before you can comment on or make changes to this bug.