Bug 2083480

Description of problem:
install kernel to test server then modify the kernel boot order, the OS can not boot the new kernel after reboot. 

Version-Release number of selected component (if applicable):
5.14.0-85.el9.x86_64
grubby-8.40-55.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. install rhel9 
2. install new kernel which the version is lower than the default one
3. update the kernel boot order with grubby 


[root@storageqe-26 ~]# uname -a
Linux storageqe-26.sqe.lab.eng.bos.redhat.com 5.14.0-85.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 4 10:53:44 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux


install new kernel kernel-5.14.0-84.el9


[root@storageqe-26 ~]# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.14.0-85.el9.x86_64"
args="ro loglevel=5 resume=/dev/mapper/rhel_storageqe--26-swap rd.lvm.lv=rhel_storageqe-26/root rd.lvm.lv=rhel_storageqe-26/swap console=ttyS1,115200 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G"
root="/dev/mapper/rhel_storageqe--26-root"
initrd="/boot/initramfs-5.14.0-85.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-85.el9.x86_64) 9.1 (Plow)"
id="3cd05b1f48494611ba7d88a96183fe48-5.14.0-85.el9.x86_64"
index=1
kernel="/boot/vmlinuz-5.14.0-84.el9.x86_64"
args="ro loglevel=5 resume=/dev/mapper/rhel_storageqe--26-swap rd.lvm.lv=rhel_storageqe-26/root rd.lvm.lv=rhel_storageqe-26/swap console=ttyS1,115200 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G"
root="/dev/mapper/rhel_storageqe--26-root"
initrd="/boot/initramfs-5.14.0-84.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-84.el9.x86_64) 9.1 (Plow)"
id="3cd05b1f48494611ba7d88a96183fe48-5.14.0-84.el9.x86_64"
index=2
kernel="/boot/vmlinuz-0-rescue-3cd05b1f48494611ba7d88a96183fe48"
args="ro loglevel=5 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_storageqe--26-swap rd.lvm.lv=rhel_storageqe-26/root rd.lvm.lv=rhel_storageqe-26/swap console=ttyS1,115200"
root="/dev/mapper/rhel_storageqe--26-root"
initrd="/boot/initramfs-0-rescue-3cd05b1f48494611ba7d88a96183fe48.img"
title="Red Hat Enterprise Linux (0-rescue-3cd05b1f48494611ba7d88a96183fe48) 9.1 (Plow)"
id="3cd05b1f48494611ba7d88a96183fe48-0-rescue"
[root@storageqe-26 ~]# 
[root@storageqe-26 ~]# grubby --set-default=/boot/vmlinuz-5.14.0-84.el9.x86_64 
The default is /boot/loader/entries/3cd05b1f48494611ba7d88a96183fe48-5.14.0-84.el9.x86_64.conf with index 1 and kernel /boot/vmlinuz-5.14.0-84.el9.x86_64
[root@storageqe-26 ~]# 
[root@storageqe-26 ~]# grubby --default-kernel
/boot/vmlinuz-5.14.0-84.el9.x86_64
[root@storageqe-26 ~]# rhts-reboot

after reboot, the OS will boot to kernel-5.14.0-85.el9.x86_64 and can not boot to kernel-5.14.0-84.el9.x86_64.

[root@storageqe-26 ~]# uname -a
Linux storageqe-26.sqe.lab.eng.bos.redhat.com 5.14.0-85.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 4 10:53:44 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@storageqe-26 ~]# 


Actual results:
can not boot to new kernel even if have set the default kernel with grubby

Expected results:
boot to the default kernel which we set

Additional info:

for now, I just hit the bug with special test server 
https://beaker.engineering.redhat.com/view/storageqe-26.sqe.lab.eng.bos.redhat.com#details