Bug 1727859

Summary: Failed to boot after upgrading a host with a custom kernel
Product: Red Hat Enterprise Virtualization Manager Reporter: Juan Orti <jortialc>
Component: redhat-virtualization-hostAssignee: Yuval Turgeman <yturgema>
Status: CLOSED ERRATA QA Contact: Yaning Wang <yaniwang>
Severity: high Docs Contact:
Priority: medium    
Version: 4.3.0CC: cshao, dfediuck, lsvaty, mavital, mtessun, nlevy, peyu, qiyuan, rdlugyhe, redhat-bugzilla, sbonazzo, weiwang, yaniwang, yturgema
Target Milestone: ovirt-4.3.5Keywords: ZStream
Target Release: 4.3.5Flags: peyu: testing_plan_complete+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: redhat-virtualization-host-4.3.5-20190710.2.el7_7 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-12 11:54:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
imgbased.log none

Description Juan Orti 2019-07-08 11:33:48 UTC
Created attachment 1588336 [details]
imgbased.log

Description of problem:
RHV-H host 4.3.0.5-0.20190409.0 with EFI boot

A hotfix kernel RPM was installed with "yum install kernel.rpm".

The host rebooted fine and in the new kernel.

Then, it was upgraded to rhvh-4.3.0.8-0.20190610.0, but the boot fails. There is only two options in grub, none of them the hotfix kernel.

rhvh-4.3.0.8-0.20190610.0
rhvh-4.3.0.5-0.20190409.0


Version-Release number of selected component (if applicable):
imgbased-1.1.7-0.1.el7ev.noarch
redhat-release-virtualization-host-4.3-0.5.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install rhvh-4.3.0.5-0.20190409.0 on EFI host
2. Install hotfix kernel with "yum install kernel.rpm" and reboot
3. Upgrade to latest RHVH and reboot

Actual results:

1. Install host
# uname -a
Linux rhvh4 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep linuxefi /boot/efi/EFI/redhat/grub.cfg 
	linuxefi /rhvh-4.3.0.5-0.20190409.0+1/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/rhvh/rhvh-4.3.0.5-0.20190409.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.0.5-0.20190409.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.0.5-0.20190409.0+1

2. Install hotfix kernel
# yum install kernel-3.10.0-957.29.1.el7.x86_64.rpm
# reboot
# uname -a
Linux rhvh4 3.10.0-957.29.1.el7.x86_64 #1 SMP Sat Jun 29 04:11:34 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep linuxefi /boot/efi/EFI/redhat/grub.cfg 
	linuxefi /rhvh-4.3.0.5-0.20190409.0+1/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/rhvh/rhvh-4.3.0.5-0.20190409.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.0.5-0.20190409.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.0.5-0.20190409.0+1
# ls  /var/imgbased/persisted-rpms/
kernel-3.10.0-957.29.1.el7.x86_64.rpm

3. Upgrade host (to redhat-virtualization-host-image-update-4.3.4-20190620.3.el7_6)
# yum update
# reboot
Fails to boot, and in the emergency shell:
# nodectl check
Status: FAILED
Bootloader ... FAILED - It looks like there are no valid bootloader entries. Please ensure this is fixed before rebooting.
  Layer boot entries ... FAILED - No bootloader entries which point to imgbased layers
  Valid boot entries ... OK
[...]
# uname -a 
Linux rhvh4 3.10.0-957.21.3.el7.x86_64 #1 SMP Fri Jun 14 02:54:29 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
# ls  /var/imgbased/persisted-rpms/
kernel-3.10.0-957.29.1.el7.x86_64.rpm

4. Reboot into old image
# uname -a
Linux rhvh4 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# grep linuxefi /boot/efi/EFI/redhat/grub.cfg 
	linuxefi /rhvh-4.3.4.1-0.20190620.0+1/vmlinuz-3.10.0-957.21.3.el7.x86_64 root=/dev/rhvh/rhvh-4.3.4.1-0.20190620.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.4.1-0.20190620.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.4.1-0.20190620.0+1
	linuxefi /rhvh-4.3.0.5-0.20190409.0+1/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/rhvh/rhvh-4.3.0.5-0.20190409.0+1 ro rd.lvm.lv=rhvh/rhvh-4.3.0.5-0.20190409.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.3.0.5-0.20190409.0+1


Expected results:
After installing the hotfix kernel and reboot, grub.cfg strangely reverts to only the old kernel. That's unexpected, a second reboot leads to booting the old kernel.
After upgrading the rhvh image, the system fails to boot, stopping at the emergency shell.
If Booting back into the old image, the hotfix kernel is lost.


Additional info:

Comment 1 Yaning Wang 2019-07-11 07:39:16 UTC
Can reproduce:

Steps to Reproduce:
1. Install rhvh-4.3.0.5-0.20190409.0 on EFI host
2. Install hotfix kernel with "yum install kernel-3.10.0-957.29.1.el7.x86_64.rpm" and reboot
3. Upgrade to redhat-virtualization-host-image-update-4.3.4-20190620.3.el7_6 and reboot


Actual results:

Fails to boot, and in the emergency shell:
# nodectl check
Status: FAILED
Bootloader ... FAILED - It looks like there are no valid bootloader entries. Please ensure this is fixed before rebooting.
  Layer boot entries ... FAILED - No bootloader entries which point to imgbased layers

Comment 3 Yaning Wang 2019-07-15 08:55:13 UTC
Steps to Reproduce:
1. Install rhvh-4.3.0.5-0.20190409.0 on EFI host
2. Install hotfix kernel with "yum install kernel-3.10.0-957.29.1.el7.x86_64.rpm" and reboot
3. Upgrade to redhat-virtualization-host-image-update-4.3.5-20190710.2 and reboot


Actual results:

1. upgrade successful
2. all 3 boot entries(the hot-fix kernel included) display correctly

Comment 6 errata-xmlrpc 2019-08-12 11:54:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2437

Comment 7 Daniel Gur 2019-08-28 13:12:16 UTC
sync2jira

Comment 8 Daniel Gur 2019-08-28 13:16:29 UTC
sync2jira