Bug 2214878

Summary: grub2 doesn't agree with grubby on default kernel
Product: [Fedora] Fedora Reporter: Aaron Lu <aaron.lwe>
Component: grub2Assignee: Nicolas Frayer <nfrayer>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 38CC: fmartine, lkundrak, mlewando, nfrayer, pgnet.dev, pjones, rharwood
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-15 12:43:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aaron Lu 2023-06-14 02:16:05 UTC
I've added 2 kernels that are built by myself using grubby --add-kernel and after that, I checked the default kernel and grubby says it's the Fedora installed kernel 6.3.7-200.fc38.x86_64 being the default kernel which is at index 2 but when I reboot, grub shows it is going to boot index 0 which is a self built kernel. I didn't use grub2-reboot to change the boot index before reboot.
I'm not sure if this is a problem of grubby or grub2 so apology if it is filed to the wrong component.

Reproducible: Always

Steps to Reproduce:
1. built a kernel by self and use grubby to install it with --add-kernel
2. grubby --info=ALL shows all kernels with the self built kernels at index 0 and index 1 and Fedora's kernel at index 2.
3. grubby --default-kernel shows Fedora's kernel /root/boot/vmlinuz-6.3.7-200.fc38.x86_64 as the default kernel, and --default-index also shows 2.
4. reboot
Actual Results:  
grub2 boot menu shows index 0 as highlighted and boot it, instead of the default index 2.

Expected Results:  
grub2 boots the default index 2 kernel.

$ sudo grubby --info=ALL
index=0
kernel="/root/boot/vmlinuz-6.4.0-rc4-00145-ga261567a74fb-dirty"
args="ro rootflags=subvol=root rhgb quiet selinux=0 sched_verbose no_hash_pointers mitigations=off"
root="UUID=36fbe348-5fae-4505-82b3-c76aa398436e"
initrd="/root/boot/initramfs-6.4.0-rc4-00145-ga261567a74fb-dirty.img"
title="6.4.0-rc4-00145-ga261567a74fb-dirty"
id="33d2bb5617aa4559a2dd6dc4f0668fac-6.4.0-rc4-00145-ga261567a74fb-dirty"
index=1
kernel="/root/boot/vmlinuz-6.4.0-rc4-00145-ga261567a74fb"
args="ro rootflags=subvol=root rhgb quiet selinux=0 sched_verbose no_hash_pointers mitigations=off"
root="UUID=36fbe348-5fae-4505-82b3-c76aa398436e"
initrd="/root/boot/initramfs-6.4.0-rc4-00145-ga261567a74fb.img"
title="6.4.0-rc4-00145-ga261567a74fb"
id="33d2bb5617aa4559a2dd6dc4f0668fac-6.4.0-rc4-00145-ga261567a74fb"
index=2
kernel="/root/boot/vmlinuz-6.3.7-200.fc38.x86_64"
args="ro rootflags=subvol=root rhgb quiet selinux=0"
root="UUID=36fbe348-5fae-4505-82b3-c76aa398436e"
initrd="/root/boot/initramfs-6.3.7-200.fc38.x86_64.img"
title="Fedora Linux (6.3.7-200.fc38.x86_64) 38 (Thirty Eight)"
id="33d2bb5617aa4559a2dd6dc4f0668fac-6.3.7-200.fc38.x86_64" 
index=3
kernel="/root/boot/vmlinuz-0-rescue-33d2bb5617aa4559a2dd6dc4f0668fac"
args="ro rootflags=subvol=root rhgb quiet selinux=0"
root="UUID=36fbe348-5fae-4505-82b3-c76aa398436e"
initrd="/root/boot/initramfs-0-rescue-33d2bb5617aa4559a2dd6dc4f0668fac.img"
title="Fedora Linux (0-rescue-33d2bb5617aa4559a2dd6dc4f0668fac) 38 (Thirty Eight)"
id="33d2bb5617aa4559a2dd6dc4f0668fac-0-rescue"

$ sudo grubby --default-kernel
/root/boot/vmlinuz-6.3.7-200.fc38.x86_64

$ sudo grubby --default-index
2

Comment 1 Aaron Lu 2023-06-14 02:46:09 UTC
In case it is useful, this is the content of my /etc/default/grub:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet selinux=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

Comment 2 Marta Lewandowska 2023-06-14 10:08:13 UTC
Hi,
Is your system BIOS or UEFI?

Could you please show the results of:
sudo ls -l /boot/grub2/
sudo ls -l /boot/efi/EFI/fedora
ls -l /etc/grub*cfg
sudo grub2-editenv - list

Comment 3 Aaron Lu 2023-06-15 01:41:45 UTC
It's UEFI.

$ sudo ls -l /boot/grub2/
total 12
drwx------. 1 root root   22 Jun  7 09:47 fonts
-rwx------. 1 root root 6876 Jun 14 15:40 grub.cfg
-rw-------  1 root root 1024 Jun 14 20:07 grubenv

$ sudo ls -l /boot/efi/EFI/fedora
total 6150
-rwx------ 1 root root     110 Jul  8  2022 BOOTX64.CSV
-rwx------ 1 root root     154 Jun  7 09:49 grub.cfg
-rwx------ 1 root root    7395 Jun  7 09:48 grub.cfg.rpmsave
-rwx------ 1 root root 3530048 Apr 12 08:00 grubx64.efi
-rwx------ 1 root root  857248 Jul  8  2022 mmx64.efi
-rwx------ 1 root root  946712 Jul  8  2022 shim.efi
-rwx------ 1 root root  946712 Jul  8  2022 shimx64.efi

$ ls -l /etc/grub*cfg
lrwxrwxrwx. 1 root root 22 Apr 12 08:00 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
lrwxrwxrwx. 1 root root 22 Apr 12 08:00 /etc/grub2-efi.cfg -> ../boot/grub2/grub.cfg

$ sudo grub2-editenv - list
blsdir=/root/boot/loader/entries
saved_entry=33d2bb5617aa4559a2dd6dc4f0668fac-6.3.7-200.fc38.x86_64
boot_success=1
next_entry=0

Comment 4 Aaron Lu 2023-06-15 02:24:58 UTC
So it appears next_entry=0 is the problem.

I tried on another box which runs Fedora 37, and "grub2-editenv - list" doesn't have next_entry=0:
$ sudo grub2-editenv - list
saved_entry=024fa9f01dc84de8b33fee4ab71977ec-6.3.7-100.fc37.x86_64
boot_success=1
boot_indeterminate=0
next_entry=

I have no idea why next_entry is set to 0 on this Fedora 38 box.

Comment 5 Marta Lewandowska 2023-06-15 07:00:58 UTC
next_entry gets set by grub2-reboot, but if you didn't run that command, then I don't know how it got there. :)
Feel free to just 
$ sudo grub2-editenv - unset next_entry
and then the whole line will just disappear from grubenv

I understand that it's working now?

Comment 6 Aaron Lu 2023-06-15 08:18:28 UTC
Hi Marta,

Thanks for the info.
Using "sudo grub2-editenv - unset next_entry" indeed clears next_entry from grub2 env and the next boot will boot the real default kernel.

The problem is, if I ever did "grub2-reboot X" for once, then the next_entry=X will be in grub2 env forever. I just tried again with "grub2-reboot 1" and after reboot, the "next_entry=1" is still there. So something seems to be wrong with the clear/unset of next_entry. Do you have any ideas what is the problem?

I do not have a separate /boot and just a compression enabled btrfs /. Not sure if this is related. The relevant partitions are:
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=winnt,errors=remount-ro)
/dev/sda2 on / type btrfs (rw,relatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root)
/dev/sda3 on /home type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Thanks.

Comment 7 Aaron Lu 2023-06-15 08:20:30 UTC
(In reply to Marta Lewandowska from comment #5)
> next_entry gets set by grub2-reboot, but if you didn't run that command,
> then I don't know how it got there. :)

So with the current finding, I guess I did "grub2-reboot 0" sometime earlier and then "next_entry=0" gets stuck in grub2 env forever :)

Comment 8 Marta Lewandowska 2023-06-15 12:02:14 UTC
What version of grub2 do you have installed? I tried installing f38 (grub2-2.06-95.fc38) and it works for me. After reboot next_entry gets unset:
# grub2-editenv - list
saved_entry=92be1469966e45b780d3652472594de0-6.2.9-300.fc38.x86_64
boot_success=0
next_entry=

Comment 9 Aaron Lu 2023-06-15 12:18:31 UTC
Hi Marta,

I have the same grub2 version:
$ rpm -qa |grep grub2
grub2-common-2.06-95.fc38.noarch
grub2-tools-minimal-2.06-95.fc38.x86_64
grub2-tools-2.06-95.fc38.x86_64
grub2-efi-x64-2.06-95.fc38.x86_64

I found this: https://discussion.fedoraproject.org/t/configuring-grub2-to-list-latest-kernels-on-boot/74831/4
It explained grub2 can not write to a btrfs partition so it can not update grubenv file. I think this explained why next_entry always get set once set on my machine since I have a single btrfs / and not a separate /boot using ext4(I should have done this :-).

Feel free to close this bug. Thanks again for your hint on the grub env.

Comment 10 Marta Lewandowska 2023-06-15 12:43:21 UTC
Aaron, thanks for sharing that.
I'm glad I could help a bit. At least you'll know what to do in the future. ;)