Bug 2157947

Summary: [leapp] IPU 7->8 on Z: grubby fails to list existing entries
Product: Red Hat Enterprise Linux 7 Reporter: Christophe Besson <cbesson>
Component: grubbyAssignee: Bootloader engineering team <bootloader-eng-team>
Status: CLOSED MIGRATED QA Contact: Release Test Team <release-test-team-automation>
Severity: low Docs Contact:
Priority: low    
Version: 7.9CC: mlewando, pstodulk
Target Milestone: rcKeywords: MigratedToJIRA, Reproducer
Target Release: ---Flags: pm-rhel: mirror+
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-16 18:19:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christophe Besson 2023-01-03 16:50:07 UTC
Description of problem:
grubby matches none of the "configFileInfo" type and falls back to the "default config" (hence grub) when /etc/grub2*.cfg files are missing, `/etc/zipl.conf` file is present and an unexpected `/etc/grub.d` directory is there.

The bug resides in grubby, but it impacts almost only Leapp, which can possibly use something else on IBM Z systems. Feel free to change the component if needed.

Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.17.0-1.el7_9.noarch

How reproducible:
Always

Steps to Reproduce:
On s390x, just create the problematic directory
# mkdir /etc/grub.d

To reproduce it more easily from an x86_64 system
# mv /etc/grub2*.cfg /tmp/
# touch /etc/zipl.conf

Actual results:
============================================================
                           ERRORS                           
============================================================

2023-01-03 11:28:56.408140 [ERROR] Actor: source_boot_loader_scanner
Message: Failed to call `grubby` to list available boot entries.
Summary:
    Details: Command ['grubby', '--info', 'ALL'] failed with exit code 1.
    Stderr: error opening /boot/grub/grub.cfg for read: No such file or directory

============================================================
                       END OF ERRORS                        
============================================================

Additional info:
* in the customer environment the directory `/etc/grub.d` was empty, hence just removing it fixed the issue.

* checking from gdb

(gdb) b readConfig
Breakpoint 1 at 0x4029b2: file grubby.c, line 1364.
(gdb) b grubby.c:5187
Breakpoint 2 at 0x404c46: file grubby.c, line 5187.

(gdb) r
Starting program: /usr/sbin/grubby --info ALL

Breakpoint 2, main (argc=<optimized out>, argv=0x7fffffffe468) at grubby.c:5187
5187			if (!grubConfig)

(gdb) list
5182		}
5183	
5184		if (!grubConfig) {
5185			if (cfi->findConfig)
5186				grubConfig = cfi->findConfig(cfi);
5187			if (!grubConfig)
5188				grubConfig = cfi->defaultConfig;
5189		}
5190	
5191		if (bootloaderProbe

(gdb) n
5186				grubConfig = cfi->findConfig(cfi);
(gdb) n
5187			if (!grubConfig)
(gdb) n
5191		if (bootloaderProbe

(gdb) p grubConfig
$1 = 0x40c307 "/boot/grub/grub.cfg"

Comment 3 Petr Stodulka 2023-01-03 17:23:49 UTC
Hi Chris, thank you for the report. It's primarily bug of the grubby component, but I understand we cannot expect it will be fixed in grubby on RHEL 7. In this case, I think we could provide a better error msg in this case to suggest the removal of the problematic directory (when exists) on s390x.

Comment 5 Marta Lewandowska 2023-01-09 15:49:25 UTC
I can easily reproduce this on s390 (or x86):

[root@ibm-z-112 ~]# mkdir /etc/grub.d
[root@ibm-z-112 ~]# grubby --info ALL
error opening /boot/grub/grub.cfg for read: No such file or directory
[root@ibm-z-112 ~]# echo $?
1

[root@ibm-z-112 ~]# rmdir /etc/grub.d
[root@ibm-z-112 ~]# grubby --info ALL
index=0
kernel=/boot/vmlinuz-3.10.0-1160.81.1.el7.s390x
args="crashkernel=auto rd.dasd=0.0.0120 rd.dasd=0.0.0121 rd.dasd=0.0.0122 rd.dasd=0.0.0123 rd.lvm.lv=rhel_ibm-z-112/root rd.lvm.lv=rhel_ibm-z-112/swap cio_ignore=all,!condev rd.znet=qeth,0.0.8000,0.0.8001,0.0.8002,layer2=1,portno=0 LANG=en_US.UTF-8"
root=/dev/mapper/rhel_ibm--z--112-root
initrd=/boot/initramfs-3.10.0-1160.81.1.el7.s390x.img
title=linux
index=1
kernel=/boot/vmlinuz-0-rescue-54ea2d4ca13940d796267154a650f8ad
args="crashkernel=auto rd.dasd=0.0.0120 rd.dasd=0.0.0121 rd.dasd=0.0.0122 rd.dasd=0.0.0123 rd.lvm.lv=rhel_ibm-z-112/root rd.lvm.lv=rhel_ibm-z-112/swap cio_ignore=all,!condev rd.znet=qeth,0.0.8000,0.0.8001,0.0.8002,layer2=1,portno=0"
root=/dev/mapper/rhel_ibm--z--112-root
initrd=/boot/initramfs-0-rescue-54ea2d4ca13940d796267154a650f8ad.img
title=linux-0-rescue-54ea2d4ca13940d796267154a650f8ad

But since the zipl bootloader, and not grub2, is used on s390, there should not be an /etc/grub.d directory in the first place. Your customer found one, but since it was empty and could be removed with no repercussions, this is indeed very low priority... especially since it's RHEL7.

Comment 6 Christophe Besson 2023-01-09 15:57:10 UTC
Yep, I didn't request a fix for RHEL 7.
Not sure what created that directory on the customer system (maybe a tuned profile or something of that kind).
I made this report essentially to prevent this issue during the IPU.

I wrote a small KCS to explain the issue in the leapp context, it could be enough...

Comment 7 RHEL Program Management 2023-09-16 18:16:06 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 8 RHEL Program Management 2023-09-16 18:19:21 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.