Bug 1990374
| Summary: | no grubenv file in /boot/efi/EFI/redhat in rhel9 arm vm | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | megao | |
| Component: | osbuild-composer | Assignee: | Ondřej Budai <obudai> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Wei Shi <wshi> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 9.0 | CC: | bootloader-eng-team, ckellner, fmartine, jbastian, linl, megao, obudai, pvlasin, ribarry, tgunders, thozza, vkuznets, wshi, xiliang, ymao | |
| Target Milestone: | beta | Keywords: | Triaged | |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | aarch64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2003038 (view as bug list) | Environment: | ||
| Last Closed: | 2021-12-07 21:30:47 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2003038 | |||
|
Description
megao
2021-08-05 09:42:32 UTC
(In reply to megao from comment #0) > Description of problem: > After booting rhel-guest-image-9.0-20210729.2.aarch64.qcow2 as a kvm guest > and installing kernel-debug.Found grubby cannot change its boot order by > using 'grubby --set-default-index=$index'. > And found the following error when booting. > error: ../../grub-core/fs/fshelp.c:257:file `/EFI/redhat/grubenv' not found > > Version-Release number of selected component (if applicable): > > RHEL Version: > RHEL-9.0(5.14.0-0.rc3.29.el9.aarch64) > can you please share the output of ls -l /boot/grub2/grubenv ? and also the output of tree /boot ? This issue exists since rhel-guest-image-8.3 from the following patch I guess, so did the osbuild image nowadays, and not sure why x86_64 images can workaround it, I didn't cover this scenario on aarch64 since we test this patch. * 6016a16 2020-07-17 | RHEL-guest-image: make the hybrid image GRUB config more reliable [Javier Martinez Canillas] [root@RHEL ~]# ls -l /boot/grub2/grubenv -rwx------. 1 root root 1024 Aug 19 07:22 /boot/grub2/grubenv [root@RHEL ~]# tree /boot /boot |-- System.map-5.14.0-0.rc4.35.el9.1.aarch64 |-- System.map-5.14.0-0.rc4.35.el9.aarch64 |-- config-5.14.0-0.rc4.35.el9.1.aarch64 |-- config-5.14.0-0.rc4.35.el9.aarch64 |-- dtb -> dtb-5.14.0-0.rc4.35.el9.aarch64 |-- dtb-5.14.0-0.rc4.35.el9.1.aarch64 | |-- amd | | |-- amd-overdrive-rev-b0.dtb | | |-- amd-overdrive-rev-b1.dtb | | |-- amd-overdrive.dtb | | `-- husky.dtb | |-- apm | | |-- apm-merlin.dtb | | `-- apm-mustang.dtb | |-- arm | | |-- foundation-v8-gicv3-psci.dtb | | |-- foundation-v8-gicv3.dtb | | |-- foundation-v8-psci.dtb | | |-- foundation-v8.dtb | | |-- fvp-base-revc.dtb | | |-- juno-r1.dtb | | |-- juno-r2.dtb | | |-- juno.dtb | | |-- rtsm_ve-aemv8a.dtb | | `-- vexpress-v2f-1xv7-ca53x2.dtb | |-- broadcom | | |-- northstar2 | | | |-- ns2-svk.dtb | | | `-- ns2-xmc.dtb | | `-- stingray | | |-- bcm958742k.dtb | | |-- bcm958742t.dtb | | `-- bcm958802a802x.dtb | |-- cavium | | |-- thunder-88xx.dtb | | `-- thunder2-99xx.dtb | |-- hisilicon | | |-- hi3660-hikey960.dtb | | |-- hi3670-hikey970.dtb | | |-- hi3798cv200-poplar.dtb | | |-- hi6220-hikey.dtb | | |-- hip05-d02.dtb | | |-- hip06-d03.dtb | | `-- hip07-d05.dtb | `-- qcom | |-- apq8016-sbc.dtb | |-- apq8094-sony-xperia-kitakami-karin_windy.dtb | |-- apq8096-db820c.dtb | |-- apq8096-ifc6640.dtb | |-- ipq6018-cp01-c1.dtb | |-- ipq8074-hk01.dtb | |-- ipq8074-hk10-c1.dtb | |-- ipq8074-hk10-c2.dtb | |-- msm8916-alcatel-idol347.dtb | |-- msm8916-asus-z00l.dtb | |-- msm8916-huawei-g7.dtb | |-- msm8916-longcheer-l8150.dtb | |-- msm8916-longcheer-l8910.dtb | |-- msm8916-mtp.dtb | |-- msm8916-samsung-a3u-eur.dtb | |-- msm8916-samsung-a5u-eur.dtb | |-- msm8992-bullhead-rev-101.dtb | |-- msm8992-msft-lumia-octagon-talkman.dtb | |-- msm8992-xiaomi-libra.dtb | |-- msm8994-angler-rev-101.dtb | |-- msm8994-msft-lumia-octagon-cityman.dtb | |-- msm8994-sony-xperia-kitakami-ivy.dtb | |-- msm8994-sony-xperia-kitakami-karin.dtb | |-- msm8994-sony-xperia-kitakami-satsuki.dtb | |-- msm8994-sony-xperia-kitakami-sumire.dtb | |-- msm8994-sony-xperia-kitakami-suzuran.dtb | |-- msm8996-mtp.dtb | |-- msm8998-asus-novago-tp370ql.dtb | |-- msm8998-hp-envy-x2.dtb | |-- msm8998-lenovo-miix-630.dtb | |-- msm8998-mtp.dtb | |-- msm8998-oneplus-cheeseburger.dtb | |-- msm8998-oneplus-dumpling.dtb | |-- qcs404-evb-1000.dtb | |-- qcs404-evb-4000.dtb | |-- qrb5165-rb5.dtb | |-- sa8155p-adp.dtb | |-- sc7180-idp.dtb | |-- sc7180-trogdor-coachz-r1-lte.dtb | |-- sc7180-trogdor-coachz-r1.dtb | |-- sc7180-trogdor-coachz-r3-lte.dtb | |-- sc7180-trogdor-coachz-r3.dtb | |-- sc7180-trogdor-lazor-limozeen-nots-r4.dtb | |-- sc7180-trogdor-lazor-limozeen-nots.dtb | |-- sc7180-trogdor-lazor-limozeen.dtb | |-- sc7180-trogdor-lazor-r0.dtb | |-- sc7180-trogdor-lazor-r1-kb.dtb | |-- sc7180-trogdor-lazor-r1-lte.dtb | |-- sc7180-trogdor-lazor-r1.dtb | |-- sc7180-trogdor-lazor-r3-kb.dtb | |-- sc7180-trogdor-lazor-r3-lte.dtb | |-- sc7180-trogdor-lazor-r3.dtb | |-- sc7180-trogdor-pompom-r1-lte.dtb | |-- sc7180-trogdor-pompom-r1.dtb | |-- sc7180-trogdor-pompom-r2-lte.dtb | |-- sc7180-trogdor-pompom-r2.dtb | |-- sc7180-trogdor-pompom-r3-lte.dtb | |-- sc7180-trogdor-pompom-r3.dtb | |-- sc7180-trogdor-r1-lte.dtb | |-- sc7180-trogdor-r1.dtb | |-- sc7280-idp.dtb | |-- sdm630-sony-xperia-ganges-kirin.dtb | |-- sdm630-sony-xperia-nile-discovery.dtb | |-- sdm630-sony-xperia-nile-pioneer.dtb | |-- sdm630-sony-xperia-nile-voyager.dtb | |-- sdm636-sony-xperia-ganges-mermaid.dtb | |-- sdm660-xiaomi-lavender.dtb | |-- sdm845-cheza-r1.dtb | |-- sdm845-cheza-r2.dtb | |-- sdm845-cheza-r3.dtb | |-- sdm845-db845c.dtb | |-- sdm845-mtp.dtb | |-- sdm845-oneplus-enchilada.dtb | |-- sdm845-oneplus-fajita.dtb | |-- sdm845-xiaomi-beryllium.dtb | |-- sdm850-lenovo-yoga-c630.dtb | |-- sm8150-hdk.dtb | |-- sm8150-microsoft-surface-duo.dtb | |-- sm8150-mtp.dtb | |-- sm8150-sony-xperia-kumano-bahamut.dtb | |-- sm8150-sony-xperia-kumano-griffin.dtb | |-- sm8250-hdk.dtb | |-- sm8250-mtp.dtb | |-- sm8250-sony-xperia-edo-pdx203.dtb | |-- sm8250-sony-xperia-edo-pdx206.dtb | |-- sm8350-hdk.dtb | `-- sm8350-mtp.dtb |-- dtb-5.14.0-0.rc4.35.el9.aarch64 | |-- amd | | |-- amd-overdrive-rev-b0.dtb | | |-- amd-overdrive-rev-b1.dtb | | |-- amd-overdrive.dtb | | `-- husky.dtb | |-- apm | | |-- apm-merlin.dtb | | `-- apm-mustang.dtb | |-- arm | | |-- foundation-v8-gicv3-psci.dtb | | |-- foundation-v8-gicv3.dtb | | |-- foundation-v8-psci.dtb | | |-- foundation-v8.dtb | | |-- fvp-base-revc.dtb | | |-- juno-r1.dtb | | |-- juno-r2.dtb | | |-- juno.dtb | | |-- rtsm_ve-aemv8a.dtb | | `-- vexpress-v2f-1xv7-ca53x2.dtb | |-- broadcom | | |-- northstar2 | | | |-- ns2-svk.dtb | | | `-- ns2-xmc.dtb | | `-- stingray | | |-- bcm958742k.dtb | | |-- bcm958742t.dtb | | `-- bcm958802a802x.dtb | |-- cavium | | |-- thunder-88xx.dtb | | `-- thunder2-99xx.dtb | |-- hisilicon | | |-- hi3660-hikey960.dtb | | |-- hi3670-hikey970.dtb | | |-- hi3798cv200-poplar.dtb | | |-- hi6220-hikey.dtb | | |-- hip05-d02.dtb | | |-- hip06-d03.dtb | | `-- hip07-d05.dtb | `-- qcom | |-- apq8016-sbc.dtb | |-- apq8094-sony-xperia-kitakami-karin_windy.dtb | |-- apq8096-db820c.dtb | |-- apq8096-ifc6640.dtb | |-- ipq6018-cp01-c1.dtb | |-- ipq8074-hk01.dtb | |-- ipq8074-hk10-c1.dtb | |-- ipq8074-hk10-c2.dtb | |-- msm8916-alcatel-idol347.dtb | |-- msm8916-asus-z00l.dtb | |-- msm8916-huawei-g7.dtb | |-- msm8916-longcheer-l8150.dtb | |-- msm8916-longcheer-l8910.dtb | |-- msm8916-mtp.dtb | |-- msm8916-samsung-a3u-eur.dtb | |-- msm8916-samsung-a5u-eur.dtb | |-- msm8992-bullhead-rev-101.dtb | |-- msm8992-msft-lumia-octagon-talkman.dtb | |-- msm8992-xiaomi-libra.dtb | |-- msm8994-angler-rev-101.dtb | |-- msm8994-msft-lumia-octagon-cityman.dtb | |-- msm8994-sony-xperia-kitakami-ivy.dtb | |-- msm8994-sony-xperia-kitakami-karin.dtb | |-- msm8994-sony-xperia-kitakami-satsuki.dtb | |-- msm8994-sony-xperia-kitakami-sumire.dtb | |-- msm8994-sony-xperia-kitakami-suzuran.dtb | |-- msm8996-mtp.dtb | |-- msm8998-asus-novago-tp370ql.dtb | |-- msm8998-hp-envy-x2.dtb | |-- msm8998-lenovo-miix-630.dtb | |-- msm8998-mtp.dtb | |-- msm8998-oneplus-cheeseburger.dtb | |-- msm8998-oneplus-dumpling.dtb | |-- qcs404-evb-1000.dtb | |-- qcs404-evb-4000.dtb | |-- qrb5165-rb5.dtb | |-- sa8155p-adp.dtb | |-- sc7180-idp.dtb | |-- sc7180-trogdor-coachz-r1-lte.dtb | |-- sc7180-trogdor-coachz-r1.dtb | |-- sc7180-trogdor-coachz-r3-lte.dtb | |-- sc7180-trogdor-coachz-r3.dtb | |-- sc7180-trogdor-lazor-limozeen-nots-r4.dtb | |-- sc7180-trogdor-lazor-limozeen-nots.dtb | |-- sc7180-trogdor-lazor-limozeen.dtb | |-- sc7180-trogdor-lazor-r0.dtb | |-- sc7180-trogdor-lazor-r1-kb.dtb | |-- sc7180-trogdor-lazor-r1-lte.dtb | |-- sc7180-trogdor-lazor-r1.dtb | |-- sc7180-trogdor-lazor-r3-kb.dtb | |-- sc7180-trogdor-lazor-r3-lte.dtb | |-- sc7180-trogdor-lazor-r3.dtb | |-- sc7180-trogdor-pompom-r1-lte.dtb | |-- sc7180-trogdor-pompom-r1.dtb | |-- sc7180-trogdor-pompom-r2-lte.dtb | |-- sc7180-trogdor-pompom-r2.dtb | |-- sc7180-trogdor-pompom-r3-lte.dtb | |-- sc7180-trogdor-pompom-r3.dtb | |-- sc7180-trogdor-r1-lte.dtb | |-- sc7180-trogdor-r1.dtb | |-- sc7280-idp.dtb | |-- sdm630-sony-xperia-ganges-kirin.dtb | |-- sdm630-sony-xperia-nile-discovery.dtb | |-- sdm630-sony-xperia-nile-pioneer.dtb | |-- sdm630-sony-xperia-nile-voyager.dtb | |-- sdm636-sony-xperia-ganges-mermaid.dtb | |-- sdm660-xiaomi-lavender.dtb | |-- sdm845-cheza-r1.dtb | |-- sdm845-cheza-r2.dtb | |-- sdm845-cheza-r3.dtb | |-- sdm845-db845c.dtb | |-- sdm845-mtp.dtb | |-- sdm845-oneplus-enchilada.dtb | |-- sdm845-oneplus-fajita.dtb | |-- sdm845-xiaomi-beryllium.dtb | |-- sdm850-lenovo-yoga-c630.dtb | |-- sm8150-hdk.dtb | |-- sm8150-microsoft-surface-duo.dtb | |-- sm8150-mtp.dtb | |-- sm8150-sony-xperia-kumano-bahamut.dtb | |-- sm8150-sony-xperia-kumano-griffin.dtb | |-- sm8250-hdk.dtb | |-- sm8250-mtp.dtb | |-- sm8250-sony-xperia-edo-pdx203.dtb | |-- sm8250-sony-xperia-edo-pdx206.dtb | |-- sm8350-hdk.dtb | `-- sm8350-mtp.dtb |-- efi | `-- EFI | |-- BOOT | | |-- BOOTAA64.EFI | | `-- fbaa64.efi | `-- redhat | |-- BOOTAA64.CSV | |-- fonts | |-- grub.cfg | |-- grubaa64.efi | |-- mmaa64.efi | |-- shim.efi | |-- shimaa64-redhat.efi | `-- shimaa64.efi |-- grub2 | `-- grubenv |-- initramfs-5.14.0-0.rc4.35.el9.1.aarch64.img |-- initramfs-5.14.0-0.rc4.35.el9.aarch64.img |-- loader | `-- entries | |-- 0eb66164daaa427a949c79633c4d7e92-5.14.0-0.rc4.35.el9.aarch64.conf | `-- ffffffffffffffffffffffffffffffff-5.14.0-0.rc4.35.el9.1.aarch64.conf |-- vmlinuz-5.14.0-0.rc4.35.el9.1.aarch64 `-- vmlinuz-5.14.0-0.rc4.35.el9.aarch64 29 directories, 254 files Update my last comment, after I tried 8.3 & 8.4 aarch64 rhel-guest-image, this issue only happens on rhel9.0 aarch64 rhel-guest-image. So the patch I mentioned in my last comment may not relevant. /boot/efi/EFI/redhat/grubenv exists on 8.3 & 8.4 aarch64 rhel-guest-image. (In reply to Wei Shi from comment #5) > Update my last comment, after I tried 8.3 & 8.4 aarch64 rhel-guest-image, > this issue only happens on rhel9.0 aarch64 rhel-guest-image. So the patch I > mentioned in my last comment may not relevant. > > /boot/efi/EFI/redhat/grubenv exists on 8.3 & 8.4 aarch64 rhel-guest-image. Thanks for the info. So this is only a RHEL9 issue or also present in 8.5 ? Checked on latest 8.5 & 8.6 aarch64 nightly composes, /boot/efi/EFI/redhat/grubenv does exist and grubby --set-default-index will update it. (In reply to Wei Shi from comment #7) > Checked on latest 8.5 & 8.6 aarch64 nightly composes, > /boot/efi/EFI/redhat/grubenv does exist and grubby --set-default-index will > update it. Got it. Can I get access to one of the problematic systems to troubleshoot ? I was not able to reproduce this with latest RHEL9 compose. $ rpm -qa | grep grub grub2-common-2.06~rc1-8.el9.noarch grub2-tools-minimal-2.06~rc1-8.el9.aarch64 grub2-tools-2.06~rc1-8.el9.aarch64 grubby-8.40-54.el9.aarch64 grub2-efi-aa64-2.06~rc1-8.el9.aarch64 $ grubby --set-default-index=0 The default is /boot/loader/entries/d1d1bc1755174c6293b228c1148d7547-5.14.0-0.rc4.35.el9.aarch64.conf with index 0 and kernel /boot/vmlinuz-5.14.0-0.rc4.35.el9.aarch64 $ grub2-editenv list | grep saved_entry saved_entry=d1d1bc1755174c6293b228c1148d7547-5.14.0-0.rc4.35.el9.aarch64 and on reboot GRUB doesn't show the error about missing grubenv, so this seems to be something that only happens on the aarch64 guest images. It would be good to get a confirmation about how these images are generated, since the issue seems to be in the image generation process and not really in the grub2 package. Because as mentioned there isn't a /boot/grub2/grub.cfg in the image (but there is a /boot/grub2/grubenv file), and there's a full config in /boot/efi/EFI/redhat/grub.cfg. So what happens is the following: 1) grubby sets a default entry to boot in /boot/grub2/grubenv. 2) the machine is rebooted 3) on reboot grub tries loads its config file from the path where it was booted (\EFI\redhat in the ESP) 4) that's /boot/efi/EFI/redhat/grub.cfg which tries to load the grubenv from its prefix (that's again \EFI\redhat in the ESP). 5) \EFI\redhat\grubenv does not exist, because is expected that the GRUB config in \EFI\redhat\grub.cfg will load the config from /boot/grub2/grub.cfg. What needs to be fixed is the image generation so the GRUB configuration files has the proper layout as expected for RHEL9. Christian confirmed that the image is built by osbuild and he already figured out the problem. Is that the unified GRUB config file is only done for "hybrid" images and an EFI aarch64 image is not hybrid. I'm changing this BZ component to osbuild since the issue is there. (In reply to Javier Martinez Canillas from comment #17) > Christian confirmed that the image is built by osbuild and he already > figured out the > problem. Is that the unified GRUB config file is only done for "hybrid" > images and an > EFI aarch64 image is not hybrid. So, I checked rhel-guest-image aarch64 8.2, 8.3 (imagefactory) 8.4, 8.5, 8.6 (osbuild), they all use the same layout like rhel9.0 which Javier just described. They don't hit the issue in this bug just because /boot/efi/EFI/redhat/grubenv does exists on rhel8.x aarch64 image. Personally I think we should only fix it in rhel9 and leave it alone for rhel8. > > I'm changing this BZ component to osbuild since the issue is there. Posted two patches, one for osbuild, one for composer, that should fix the issue by using the unified layout on pure EFI systems (such as aarch64) on RHEL 9. Hello Wei, can you please verify this bug? We redeployed the internal image building service today so tomorrow's nightly should contain the fix. Thanks! Ondřej I'm seeing this on bare-metal x86 UEFI systems too. I opened new bug 2007365 for it. |