Bug 1727859 - Failed to boot after upgrading a host with a custom kernel
Summary: Failed to boot after upgrading a host with a custom kernel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: redhat-virtualization-host
Version: 4.3.0
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ovirt-4.3.5
: 4.3.5
Assignee: Yuval Turgeman
QA Contact: Yaning Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-08 11:33 UTC by Juan Orti
Modified: 2022-07-09 14:15 UTC (History)
14 users (show)

Fixed In Version: redhat-virtualization-host-4.3.5-20190710.2.el7_7
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-12 11:54:28 UTC
oVirt Team: Node
Target Upstream Version:
Embargoed:
peyu: testing_plan_complete+


Attachments (Terms of Use)
imgbased.log (232.36 KB, text/plain)
2019-07-08 11:33 UTC, Juan Orti
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-38480 0 None None None 2022-07-09 14:15:12 UTC
Red Hat Knowledge Base (Solution) 4270552 0 None None None 2019-07-08 11:33:48 UTC
Red Hat Product Errata RHSA-2019:2437 0 None None None 2019-08-12 11:54:41 UTC
oVirt gerrit 101622 0 master MERGED persist: install rpms before setting up grub 2020-12-15 06:20:03 UTC
oVirt gerrit 101652 0 ovirt-4.3 MERGED persist: install rpms before setting up grub 2020-12-15 06:20:01 UTC

Description Juan Orti 2019-07-08 11:33:48 UTC
Created attachment 1588336 [details]
imgbased.log

Description of problem:
RHV-H host 4.3.0.5-0.20190409.0 with EFI boot

A hotfix kernel RPM was installed with "yum install kernel.rpm".

The host rebooted fine and in the new kernel.

Then, it was upgraded to rhvh-4.3.0.8-0.20190610.0, but the boot fails. There is only two options in grub, none of them the hotfix kernel.

rhvh-4.3.0.8-0.20190610.0
rhvh-4.3.0.5-0.20190409.0


Version-Release number of selected component (if applicable):
imgbased-1.1.7-0.1.el7ev.noarch
redhat-release-virtualization-host-4.3-0.5.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install rhvh-4.3.0.5-0.20190409.0 on EFI host
2. Install hotfix kernel with "yum install kernel.rpm" and reboot
3. Upgrade to latest RHVH and reboot

Actual results:

1. Install host
# uname -a
Linux rhvh4 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep linuxefi /boot/efi/EFI/redhat/grub.cfg 
	linuxefi /rhvh-4.3.0.5-0.20190409.0+1/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/rhvh/rhvh-4.3.0.5-0.20190409.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.0.5-0.20190409.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.0.5-0.20190409.0+1

2. Install hotfix kernel
# yum install kernel-3.10.0-957.29.1.el7.x86_64.rpm
# reboot
# uname -a
Linux rhvh4 3.10.0-957.29.1.el7.x86_64 #1 SMP Sat Jun 29 04:11:34 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep linuxefi /boot/efi/EFI/redhat/grub.cfg 
	linuxefi /rhvh-4.3.0.5-0.20190409.0+1/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/rhvh/rhvh-4.3.0.5-0.20190409.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.0.5-0.20190409.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.0.5-0.20190409.0+1
# ls  /var/imgbased/persisted-rpms/
kernel-3.10.0-957.29.1.el7.x86_64.rpm

3. Upgrade host (to redhat-virtualization-host-image-update-4.3.4-20190620.3.el7_6)
# yum update
# reboot
Fails to boot, and in the emergency shell:
# nodectl check
Status: FAILED
Bootloader ... FAILED - It looks like there are no valid bootloader entries. Please ensure this is fixed before rebooting.
  Layer boot entries ... FAILED - No bootloader entries which point to imgbased layers
  Valid boot entries ... OK
[...]
# uname -a 
Linux rhvh4 3.10.0-957.21.3.el7.x86_64 #1 SMP Fri Jun 14 02:54:29 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
# ls  /var/imgbased/persisted-rpms/
kernel-3.10.0-957.29.1.el7.x86_64.rpm

4. Reboot into old image
# uname -a
Linux rhvh4 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep linuxefi /boot/efi/EFI/redhat/grub.cfg 
	linuxefi /rhvh-4.3.4.1-0.20190620.0+1/vmlinuz-3.10.0-957.21.3.el7.x86_64 root=/dev/rhvh/rhvh-4.3.4.1-0.20190620.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.4.1-0.20190620.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.4.1-0.20190620.0+1
	linuxefi /rhvh-4.3.0.5-0.20190409.0+1/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/rhvh/rhvh-4.3.0.5-0.20190409.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.0.5-0.20190409.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.0.5-0.20190409.0+1


Expected results:
After installing the hotfix kernel and reboot, grub.cfg strangely reverts to only the old kernel. That's unexpected, a second reboot leads to booting the old kernel.
After upgrading the rhvh image, the system fails to boot, stopping at the emergency shell.
If Booting back into the old image, the hotfix kernel is lost.


Additional info:

Comment 1 Yaning Wang 2019-07-11 07:39:16 UTC
Can reproduce:

Steps to Reproduce:
1. Install rhvh-4.3.0.5-0.20190409.0 on EFI host
2. Install hotfix kernel with "yum install kernel-3.10.0-957.29.1.el7.x86_64.rpm" and reboot
3. Upgrade to redhat-virtualization-host-image-update-4.3.4-20190620.3.el7_6 and reboot


Actual results:

Fails to boot, and in the emergency shell:
# nodectl check
Status: FAILED
Bootloader ... FAILED - It looks like there are no valid bootloader entries. Please ensure this is fixed before rebooting.
  Layer boot entries ... FAILED - No bootloader entries which point to imgbased layers

Comment 3 Yaning Wang 2019-07-15 08:55:13 UTC
Steps to Reproduce:
1. Install rhvh-4.3.0.5-0.20190409.0 on EFI host
2. Install hotfix kernel with "yum install kernel-3.10.0-957.29.1.el7.x86_64.rpm" and reboot
3. Upgrade to redhat-virtualization-host-image-update-4.3.5-20190710.2 and reboot


Actual results:

1. upgrade successful
2. all 3 boot entries(the hot-fix kernel included) display correctly

Comment 6 errata-xmlrpc 2019-08-12 11:54:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2437

Comment 7 Daniel Gur 2019-08-28 13:12:16 UTC
sync2jira

Comment 8 Daniel Gur 2019-08-28 13:16:29 UTC
sync2jira


Note You need to log in before you can comment on or make changes to this bug.