What does grubby --default-kernel report?
Is this UEFI or BIOS?

(In reply to Marta Lewandowska from comment #1)
> What does grubby --default-kernel report?

# grubby --default-kernel
/boot/vmlinuz-5.14.0-162.el9.x86_64
but the kernel booted is
# uname -r
5.14.0+

> Is this UEFI or BIOS?

UEFI

Thanks for the kernel. When I first reinstall grub2-common and then install your centos kernel, it works for me. The grub menu looks the same as yours, grubby reports the same:
[root@hpe-dl360gen10-01 kernel]# grubby --default-kernel
/boot/vmlinuz-5.14.0-70.13.1.el9_0.x86_64
[root@hpe-dl360gen10-01 kernel]# grub2-editenv list
saved_entry=9dc196ee57d848d58a768eb6c156aeea-5.14.0-70.13.1.el9_0.x86_64

but the highlighted entry in the grub menu is 5.14.0-70.13.1.el9_0 and so it reboots by default to that kernel...
[root@hpe-dl360gen10-01 ~]# uname -r
5.14.0-70.13.1.el9_0.x86_64

Please let me know if this works for you too. If so, then I guess this is another symptom of: https://bugzilla.redhat.com/show_bug.cgi?id=2015825#c39

All right, I reinstalled grub2-common to get the stub, installed your centOS kernel, rebooted to -70, which was the default.
Then installed -162 from brew and rebooted.
Grub menu always has the self built kernel at the top, but for me the same kernel that grubby has for default is the one that is highlighted in the grub menu, and it is the one to which the machine reboots. I tried this also on the jacobsville machine, as you know. So... I dunno... I can't reproduce it. :(

Red Hat Enterprise Linux (5.14.0+) 9.0 (Plow)                                     
Red Hat Enterprise Linux (5.14.0-162.el9.x86_64) 9.0 (Plow)                  <--- 
Red Hat Enterprise Linux (5.14.0-70.13.1.el9_0.x86_64) 9.0 (Plow)                  
Red Hat Enterprise Linux (0-rescue-39329705e4664feaa1eb3a9391f86aba) 9.0>
UEFI Firmware Settings

[root@intel-jacobsville-01 ~]# grubby --info ALL
index=0
kernel="/boot/vmlinuz-5.14.0-162.el9.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_intel--jacobsville--01-swap rd.lvm.lv=rhel_intel-jacobsville-01/root rd.lvm.lv=rhel_intel-jacobsville-01/swap nomodeset console=ttyS0,115200n81"
root="/dev/mapper/rhel_intel--jacobsville--01-root"
initrd="/boot/initramfs-5.14.0-162.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-162.el9.x86_64) 9.0 (Plow)"
id="39329705e4664feaa1eb3a9391f86aba-5.14.0-162.el9.x86_64"
index=1
kernel="/boot/vmlinuz-5.14.0-70.13.1.el9_0.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_intel--jacobsville--01-swap rd.lvm.lv=rhel_intel-jacobsville-01/root rd.lvm.lv=rhel_intel-jacobsville-01/swap nomodeset console=ttyS0,115200n81"
root="/dev/mapper/rhel_intel--jacobsville--01-root"
initrd="/boot/initramfs-5.14.0-70.13.1.el9_0.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-70.13.1.el9_0.x86_64) 9.0 (Plow)"
id="39329705e4664feaa1eb3a9391f86aba-5.14.0-70.13.1.el9_0.x86_64"
index=2
kernel="/boot/vmlinuz-5.14.0+"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_intel--jacobsville--01-swap rd.lvm.lv=rhel_intel-jacobsville-01/root rd.lvm.lv=rhel_intel-jacobsville-01/swap nomodeset console=ttyS0,115200n81"
root="/dev/mapper/rhel_intel--jacobsville--01-root"
initrd="/boot/initramfs-5.14.0+.img"
title="Red Hat Enterprise Linux (5.14.0+) 9.0 (Plow)"
id="39329705e4664feaa1eb3a9391f86aba-5.14.0+"
index=3
kernel="/boot/vmlinuz-0-rescue-39329705e4664feaa1eb3a9391f86aba"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_intel--jacobsville--01-swap rd.lvm.lv=rhel_intel-jacobsville-01/root rd.lvm.lv=rhel_intel-jacobsville-01/swap nomodeset console=ttyS0,115200n81"
root="/dev/mapper/rhel_intel--jacobsville--01-root"
initrd="/boot/initramfs-0-rescue-39329705e4664feaa1eb3a9391f86aba.img"
title="Red Hat Enterprise Linux (0-rescue-39329705e4664feaa1eb3a9391f86aba) 9.0 (Plow)"
id="39329705e4664feaa1eb3a9391f86aba-0-rescue"
[root@intel-jacobsville-01 ~]# grubby --default-kernel
/boot/vmlinuz-5.14.0-162.el9.x86_64

[root@intel-jacobsville-01 ~]# uname -r
5.14.0-162.el9.x86_64

The problem is shown right in your example. On the serial console the first entry is
Red Hat Enterprise Linux (5.14.0+) 9.0 (Plow) 
but grubby has
index=0
kernel="/boot/vmlinuz-5.14.0-162.el9.x86_64"
while it should be
kernel="/boot/vmlinuz-5.14.0+"

The other entries don't match up either. Thanks for looking into it!

So much communication, and I clearly misunderstood the issue! ;) I thought the main problem was that grubby was showing kernelX as default, while kernelY was the one that was actually booting. This is clearly not desired, especially if the grub menu is hidden by default. But I guess you are mainly concerned with how the menu looks and the order of the entries therein...
I'm afraid that if functionality is preserved, as it appears to be, we will just have to deal with the aesthetic problem. At least for now. Sorry about that. :)

The problem is not only an aesthetic one because it also effects grubby --set-default-index=

Now if I run
# grubby --set-default-index=0
The default is /boot/loader/entries/b5f78ac9563f4b44a17a43cff8549745-5.14.0-117.el9.x86_64.conf with index 0 and kernel /boot/vmlinuz-5.14.0-117.el9.x86_64
and reboot
      Red Hat Enterprise Linux (5.14.0+) 9.0 (Plow)
      Red Hat Enterprise Linux (5.14.0-117.el9.x86_64) 9.0 (Plow) <----- boots correct entry
      Red Hat Enterprise Linux (5.14.0-70.13.1.el9_0.x86_64) 9.0 (Plow)
      UEFI Firmware Settings

However we should at least understand why this happens and fix it. This is a regression as it works as expected in RHEL8.

Well, let's back up.  Nowhere is it specified that grubby index reflects the order in grub.

grubby indexes are generated by sorting the BLS files.  They're merely a convenience so that you don't have to specify the full id.  Meanwhile, grub2 controlls its own menu ordering.  It's possible to manually edit config files and rearrange entries, for instance.

If by default they sort differently, that may be a bit odd cosmetically, but it's certainly not a regression.  "Regression" is not a word to be thrown around lightly when something changes.  15.8 is clear on this: "Changes to undocumented interfaces" are not regressions.  If you wanted to claim regression, you'd also need to claim a previously working feature that no longer works, and there isn't one here.

Hi Robbie,

I agree with you when you say there is no feature that says grubby and grub2 have to show the same boot order. I took this for granted, but maybe we could fix it anyway since I've been using this method for a long time. Thanks!

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.