Bug 2131234 - [leapp] /boot mounted too late for the "remove_boot_files" actor
Summary: [leapp] /boot mounted too late for the "remove_boot_files" actor
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: leapp-repository
Version: 7.9
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Leapp Notifications Bot
QA Contact: upgrades-and-conversions
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-30 11:34 UTC by Christophe Besson
Modified: 2023-07-31 07:12 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OAMG-7700 0 None None None 2022-09-30 11:37:33 UTC
Red Hat Issue Tracker RHELPLAN-135433 0 None None None 2022-09-30 11:37:31 UTC

Description Christophe Besson 2022-09-30 11:34:59 UTC
Description of problem:
During an IPU 7>8, customer reported RHEL 8.6 fails stalls at reboot.

Checking the logs, it appears:
 - the upgrade images have not been removed from /boot, explaining why it always reboots on this.
 - rebooting on the RHEL 7.9 kernel, the OS seems to be correctly upgraded anyway (rpms are updated and /etc/redhat-release shows 8.6).
 - no el8 kernel boot entry

The sosreport shows they have 40 disks with multipath and lpfc nics.
However sda where resides the rootfs, /boot and /boot/efi is not a mpath.

NAME                     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                        8:0    0 558.4G  0 disk  
|-sda1                     8:1    0   200M  0 part  /boot/efi
|-sda2                     8:2    0   500M  0 part  /boot
`-sda3                     8:3    0 557.7G  0 part  
  |-rhel-root            253:0    0    50G  0 lvm   /
  |-rhel-swap            253:1    0     4G  0 lvm   [SWAP]


Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.16.0-4.el7_9

How reproducible:
Always for the customer.
Cannot reproduce internally.

Actual results:
$ grep -e leapp.workflow.Preparation.remove_boot_files -e sda2 0020-leapp-logs.tar/leapp/leapp-upgrade.log | grep localhost | head -n5
Sep 20 00:56:40 localhost kernel:  sda: sda1 sda2 sda3
Sep 20 00:59:02 localhost upgrade[4157]: 2022-09-19 17:59:02.297 ERROR    PID: 862 leapp.workflow.Preparation.remove_boot_files: Could not remove /boot/vmlinuz-upgrade.x86_64: [Errno 2] No such file or directory: '/boot/vmlinuz-upgrade.x86_64'.
Sep 20 00:59:02 localhost upgrade[4157]: 2022-09-19 17:59:02.362 ERROR    PID: 862 leapp.workflow.Preparation.remove_boot_files: Could not remove /boot/initramfs-upgrade.x86_64.img: [Errno 2] No such file or directory: '/boot/initramfs-upgrade.x86_64.img'.
Sep 20 01:21:09 localhost kernel: XFS (sda2): Mounting V4 Filesystem          <===
Sep 20 01:21:21 localhost kernel: XFS (sda2): Ending clean mount

Additional info:
- Need to request an sosreport taken on the upgraded OS, with the leapp.db.
- It seems replacing the UUIDs by the device names in /etc/fstab didn't help to move forward.
- Trying to hack the leapp dracut do-upgrade script to introduce a delay and force the mount of /boot didn't help, customer reported "/dev/sda2 can't open blockdev".
- Only workaround for now: fix manually the issue after the IPU (outlines: reboot on el7 kernel, remove upgrade imgs from /boot, reinstall the el8 kernel and check the grub.cfg and bls entries are correct).


Note You need to log in before you can comment on or make changes to this bug.