Bug 1767395

Summary: grub2 is not listing and booting selected kernel
Product: [Fedora] Fedora Reporter: Jan Stancek <jstancek>
Component: grub2Assignee: Peter Jones <pjones>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: fmartine, lkundrak, pjones
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: grub2-2.04-6.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-27 13:13:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Stancek 2019-10-31 11:27:58 UTC 
Description of problem:
After make modules_install && make install, I ran "grubby --set-default".

PROBLEM: grub2 is not booting selected kernel, it boots different kernel

# cat /boot/loader/entries/6b06cfc1c3554b69a9364c22010169ac-5.3.0+.conf 
title Fedora (5.3.0+) 31 (Thirty One)
version 5.3.0+
linux /vmlinuz-5.3.0+
initrd /initramfs-5.3.0+.img
options $kernelopts
id fedora-20191031105856-5.3.0+
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel-

# grubby --set-default /boot/vmlinuz-5.3.0+ 
# reboot
...
# uname -r
5.3.7-301.fc31.x86_64

# cat /boot/grub2/grubenv
# GRUB Environment Block
saved_entry=6b06cfc1c3554b69a9364c22010169ac-5.3.0+
kernelopts=root=/dev/mapper/fedora_intel--wildcatpass--05-root ro resume=/dev/mapper/fedora_intel--wildcatpass--05-swap rd.lvm.lv=fedora_intel-wildcatpass-05/root rd.lvm.lv=fedora_intel-wildcatpass-05/swap console=ttyS0,115200n81
boot_success=0
boot_indeterminate=33
########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

grub2 menu doesn't show this entry either:
  Fedora (5.3.7-301.fc31.x86_64) 31 (Thirty One)                            
  Fedora (5.3.0-rc7+) 31 (Thirty One)                                      
  Fedora (5.3.0-rc1+) 31 (Thirty One)                                      
  Fedora (5.3.0) 31 (Thirty One)                                           
  Fedora (0-rescue-6b06cfc1c3554b69a9364c22010169ac) 31 (Thirty One)   

After I delete /boot/loader/entries/6b06cfc1c3554b69a9364c22010169ac-5.3.0.conf 
5.3.0+ starts working, it's visible in menu and grub2 boots correct kernel as expected.

It appears that combination of 5.3.0 and 5.3.0+ present at the same time is the trigger:

# cat /boot/loader/entries/6b06cfc1c3554b69a9364c22010169ac-5.3.0.conf 
title Fedora (5.3.0) 31 (Thirty One)
version 5.3.0
linux /vmlinuz-5.3.0
initrd /initramfs-5.3.0.img
options $kernelopts
id fedora-20191031092505-5.3.0
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

# cat /boot/loader/entries/6b06cfc1c3554b69a9364c22010169ac-5.3.0+.conf 
title Fedora (5.3.0+) 31 (Thirty One)
version 5.3.0+
linux /vmlinuz-5.3.0+
initrd /initramfs-5.3.0+.img
options $kernelopts
id fedora-20191031105856-5.3.0+
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel-


Version-Release number of selected component (if applicable):
grub2-pc-2.02-100.fc31.x86_64

How reproducible:
100%

Steps to Reproduce:
see above

Actual results:
grub2 is not listing and booting selected kernel

Expected results:
grub2 menu shows both 5.3.0 and 5.3.0+
grub2 boots 5.3.0+ when it's set by grubby

Additional info:
Comment 1 Javier Martinez Canillas 2019-10-31 11:30:27 UTC 
Is this with an EFI or legacy BIOS install?
Comment 2 Jan Stancek 2019-10-31 11:34:55 UTC 
legacy:

# rpm -qa grub2*
grub2-common-2.02-100.fc31.noarch
grub2-tools-2.02-100.fc31.x86_64
grub2-tools-minimal-2.02-100.fc31.x86_64
grub2-tools-extra-2.02-100.fc31.x86_64
grub2-pc-2.02-100.fc31.x86_64
grub2-pc-modules-2.02-100.fc31.noarch
Comment 3 Javier Martinez Canillas 2019-10-31 11:40:45 UTC 
The GRUB core.img is not updated on package upgrades. I believe that this issue has already been fixed but you still have an old GRUB installed.

You may try to update it with 'grub2-install /dev/sdX' (where sdX is the block device where GRUB is installed).
Comment 4 Jan Stancek 2019-10-31 12:45:10 UTC 
It's a fresh install, but I did try that anyway -> no change.

Problem appears to be that for blscfg, vercmp("5.3.0", "5.3.0+") == 0.
Whatever entry is found first makes it to menu list:
...
commands/blscfg.c:418: filename: "6b06cfc1c3554b69a9364c22010169ac-5.3.0+.conf"
commands/blscfg.c:97: new keyval at 0x78eda5d0:title:Fedora (5.3.0+) 31 (Thirty One)
commands/blscfg.c:97: new keyval at 0x78eda560:version:5.3.0+
commands/blscfg.c:97: new keyval at 0x78eda500:linux:/vmlinuz-5.3.0+
commands/blscfg.c:97: new keyval at 0x78eda4a0:initrd:/initramfs-5.3.0+.img
commands/blscfg.c:97: new keyval at 0x78eda5f0:options:$kernelopts
commands/blscfg.c:97: new keyval at 0x78eda3c0:id:fedora-20191031105856-5.3.0+
commands/blscfg.c:97: new keyval at 0x78eda350:grub_users:$grub_users
commands/blscfg.c:97: new keyval at 0x78eda2f0:grub_arg:--unrestricted
commands/blscfg.c:97: new keyval at 0x78eda430:grub_class:kernel-
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0+ and 5.3.7
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0+ and 5.3.0
commands/blscfg.c:150: vercmp comparing  and rc7+
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0+ and 5.3.0
commands/blscfg.c:150: vercmp comparing  and rc1+
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and  6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0+ and 0
commands/blscfg.c:380: Add entry with id "6b06cfc1c3554b69a9364c22010169ac-5.3.0+"

Second one is then presumably viewed as duplicate by:
  FOR_BLS_ENTRIES(e) {
    rc = bls_cmp(entry, e);
    if (!rc)
      return GRUB_ERR_BAD_ARGUMENT;

commands/blscfg.c:418: filename: "6b06cfc1c3554b69a9364c22010169ac-5.3.0.conf"
commands/blscfg.c:97: new keyval at 0x78eda0d0:title:Fedora (5.3.0) 31 (Thirty One)
commands/blscfg.c:97: new keyval at 0x78eda060:version:5.3.0
commands/blscfg.c:97: new keyval at 0x78eda000:linux:/vmlinuz-5.3.0
commands/blscfg.c:97: new keyval at 0x78ed9fa0:initrd:/initramfs-5.3.0.img
commands/blscfg.c:97: new keyval at 0x78eda0f0:options:$kernelopts
commands/blscfg.c:97: new keyval at 0x78ed9ec0:id:fedora-20191031092505-5.3.0
commands/blscfg.c:97: new keyval at 0x78ed9e50:grub_users:$grub_users
commands/blscfg.c:97: new keyval at 0x78ed9df0:grub_arg:--unrestricted
commands/blscfg.c:97: new keyval at 0x78ed9f30:grub_class:kernel
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0 and 5.3.7
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0 and 5.3.0
commands/blscfg.c:150: vercmp comparing  and rc7+
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0 and 5.3.0
commands/blscfg.c:150: vercmp comparing  and rc1+
commands/blscfg.c:150: vercmp comparing 6b06cfc1c3554b69a9364c22010169ac and 6b06cfc1c3554b69a9364c22010169ac
commands/blscfg.c:150: vercmp comparing 5.3.0 and 5.3.0+
commands/blscfg.c:150: vercmp comparing  and 

commands/blscfg.c:1022: bls_create_entries Creating entries from bls
commands/blscfg.c:713: create_entry got here
commands/blscfg.c:753: adding menu entry for "Fedora (5.3.7-301.fc31.x86_64) 31 (Thirty One)" with id "6b06cfc1c3554b69a9364c22010169ac-5.3.7-301.fc31.x86_64"
commands/blscfg.c:817: adding initrd /initramfs-5.3.7-301.fc31.x86_64.img
commands/blscfg.c:833: Added entry 0 id:"6b06cfc1c3554b69a9364c22010169ac-5.3.7-301.fc31.x86_64"
commands/blscfg.c:713: create_entry got here
commands/blscfg.c:753: adding menu entry for "Fedora (5.3.0-rc7+) 31 (Thirty One)" with id "6b06cfc1c3554b69a9364c22010169ac-5.3.0-rc7+"
commands/blscfg.c:817: adding initrd /initramfs-5.3.0-rc7+.img
commands/blscfg.c:833: Added entry 1 id:"6b06cfc1c3554b69a9364c22010169ac-5.3.0-rc7+"
commands/blscfg.c:713: create_entry got here
commands/blscfg.c:753: adding menu entry for "Fedora (5.3.0-rc1+) 31 (Thirty One)" with id "6b06cfc1c3554b69a9364c22010169ac-5.3.0-rc1+"
commands/blscfg.c:817: adding initrd /initramfs-5.3.0-rc1+.img
commands/blscfg.c:833: Added entry 2 id:"6b06cfc1c3554b69a9364c22010169ac-5.3.0-rc1+"
commands/blscfg.c:713: create_entry got here
commands/blscfg.c:753: adding menu entry for "Fedora (5.3.0+) 31 (Thirty One)" with id "6b06cfc1c3554b69a9364c22010169ac-5.3.0+"
commands/blscfg.c:817: adding initrd /initramfs-5.3.0+.img
commands/blscfg.c:833: Added entry 3 id:"6b06cfc1c3554b69a9364c22010169ac-5.3.0+"
commands/blscfg.c:713: create_entry got here
commands/blscfg.c:753: adding menu entry for "Fedora (0-rescue-6b06cfc1c3554b69a9364c22010169ac) 31 (Thirty One)" with id "6b06cfc1c3554b69a9364c22010169ac-0-rescue"
commands/blscfg.c:817: adding initrd /initramfs-0-rescue-6b06cfc1c3554b69a9364c22010169ac.img
commands/blscfg.c:833: Added entry 4 id:"6b06cfc1c3554b69a9364c22010169ac-

^^ In this example 5.3.0 entry is missing.
Comment 5 Javier Martinez Canillas 2019-10-31 15:13:13 UTC 
(In reply to Jan Stancek from comment #4)
> It's a fresh install, but I did try that anyway -> no change.
> 
> Problem appears to be that for blscfg, vercmp("5.3.0", "5.3.0+") == 0.
> Whatever entry is found first makes it to menu list:

Thanks a lot for figuring what's causing the issue. That seems to be the problem indeed.
Comment 6 Javier Martinez Canillas 2019-11-04 13:44:02 UTC 
(In reply to Jan Stancek from comment #4)
> It's a fresh install, but I did try that anyway -> no change.
> 
> Problem appears to be that for blscfg, vercmp("5.3.0", "5.3.0+") == 0.

We are using the same algorithm than rpmvercmp() to sort the boot entries and as you mentioned rpmvercmp("5.3.0", "5.3.0+") == 0.

Fixing this in the blscfg module is trivial but I'm discussing it with the rpmlib maintainers since we don't want to diverge from the RPM sorting.
Comment 7 Jan Stancek 2019-11-04 13:54:59 UTC 
Thanks for update. Main issue for me is that this currently breaks kernel bisecting - you/script run grubby, thinking you are testing a specific kernel version, but a different kernel is booted instead. Which obviously leads to surprising results.
Comment 8 Javier Martinez Canillas 2019-11-04 15:02:00 UTC 
(In reply to Jan Stancek from comment #7)
> Thanks for update. Main issue for me is that this currently breaks kernel
> bisecting - you/script run grubby, thinking you are testing a specific
> kernel version, but a different kernel is booted instead. Which obviously
> leads to surprising results.

Yes, understood. I've created an issue for rpm since this is a more generic problem of rpmvercmp() comparing as equal version strings that aren't the same.

https://github.com/rpm-software-management/rpm/issues/925

For the specific case of the + character, I've proposed a pull-request:

https://github.com/rpm-software-management/rpm/pull/924

Even if that's not accepted for rpmvercmp(), I think that we should do it for the blscfg module since it's common to have installed kernel versions with a + character (it's the default that's appended by setlocalversion if the CONFIG_LOCALVERSION_AUTO option isn't enabled).