Bug 1931443

Description of problem:
After reboot, a RHV-H host fails to boot after GRUB2 screen displaying errors.

error: ../../grub-core/loader/i386/pc/linux.c:170:invalid magic number
error: ../../grub-core/loader/i386/pc/linux.c:1418:you need to load the kernel first.

This issue occurs sometimes when a host is placed in maintenance then rebooted.
It also happens sometimes after upgrading a RHV-H host.
It does not affect all hosts in a cluster and also does not affect the same host every time. 

This issue has occurred unexpectedly in the customer environment multiple times (see linked cases) and once in my RHV 4.4.3 lab.

We still don't know the cause but it seems a task/process is truncating the initramfs and vmlinuz files in /boot and /boot/rhvh--4.4.x.x-x.YYYYMMDD.0+1 to 0 bytes.


Version-Release number of selected component (if applicable):
RHV-H 4.4.3
RHV-H 4.3.7
RHV-H 4.3.6

How reproducible:
100%

Steps to Reproduce: **only reproduces sometimes in customer environment.
1. Place host in maintenance.
2. Reboot host.
or
1. Place host in maintenance.
2. Upgrade RHV-H to newer version
3. Reboot

Steps to Reproduce: **reproduces successfully.
1. Place host in maintenance.
2. Wipe initramfs and vmlinuz files in /boot directory
[root@rhvh1 ~]# > /boot/initramfs-4.18.0-240.1.1.el8_3.x86_64.img
[root@rhvh1 ~]# > /boot/vmlinuz-4.18.0-240.1.1.el8_3.x86_64
[root@rhvh1 ~]# > /boot/rhvh-4.4.3.1-0.20201116.0+1/initramfs-4.18.0-240.1.1.el8_3.x86_64.img
[root@rhvh1 ~]# > /boot/rhvh-4.4.3.1-0.20201116.0+1/vmlinuz-4.18.0-240.1.1.el8_3.x86_64
3. Reboot host


Actual results:

After GRUB screen, the following errors are displayed.
error: ../../grub-core/loader/i386/pc/linux.c:170:invalid magic number.
error: ../../grub-core/loader/i386/pc/linux.c:1418:you need to load the kernel first.
Press any key to continue…

Host is unable to find the kernel to boot.
Pressing any key reboots the host and same errors are displayed.


Expected results:
RHV-H host should boot.


Additional info:

1. Upon booting a rescue ISO and inspecting /boot, it appears the file size for initramfs and vmlinuz files in /boot/rhvh-4.4.3.1-0.20201116.0+1/ is 0 bytes.

[root@rhvh1 ~]# ls -l /boot/rhvh-4.4.3.1-0.20201116.0+1/
total 4.1M
drwxr-xr-x. 2 root root 4.0K Feb 3 15:43 .
dr-xr-xr-x. 7 root root 4.0K Jan 6 16:24 ..
-rw-r--r--. 1 root root 186K Oct 16 19:52 config-4.18.0-240.1.1.el8_3.x86_64
-rw-------. 1 root root    0 Jan 6 16:04 initramfs-4.18.0-240.1.1.el8_3.x86_64.img
-rw-------. 1 root root 3.9M Oct 16 17:52 System.map-4.18.0-240.1.1.el8_3.x86_64
-rwxr-xr-x. 1 root root    0 Oct 16 17:52 vmlinuz-4.18.0-240.1.1.el8_3.x86_64
-rw-r--r--. 1 root root  172 Oct 16 19:51 .vmlinuz-4.18.0-240.1.1.el8_3.x86_64.hmac

Recovering from this issue requires copying initramfs and vmlinuz files from /boot to /boot/rhvh-4.4.3.1-0.20201116.0+1.

However, in some cases, the initramfs and vmlinuz files in /boot are also truncated to 0 bytes so the files need to be recovered from a backup.