1927688 – [leapp] unable to mount the partitions upon reboot on the upgrade initramfs

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1927688 - [leapp] unable to mount the partitions upon reboot on the upgrade initramfs

Summary: [leapp] unable to mount the partitions upon reboot on the upgrade initramfs

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	leapp-repository
Sub Component:
Version:	7.9
Hardware:	All
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Vinzenz Feenstra [evilissimo]
QA Contact:	Alois Mahdal
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1818088
TreeView+	depends on / blocked

Reported:	2021-02-11 10:37 UTC by Christophe Besson
Modified:	2024-12-20 19:37 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-05-03 07:49:46 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Christophe Besson 2021-02-11 10:37:21 UTC

Description of problem:
Customer ran a leapp upgrade and the first step worked fine.
Once they rebooted on the upgrade initramfs, they got a python traceback, leapp being unable to issue a `mount -a`. This is a consequence of a warning printed out just before the issue:
"""
WARNING: PV /dev/sda2 in VG rhel is using an old PV header, modify the VG to update.
"""

Version-Release number of selected component (if applicable):
leapp-repository-0.12.0-2.el7_9.noarch

How reproducible:
100% for the customer.

Steps to Reproduce:
Didn't try. The OS was installed originally with an old RHEL 7 (< 7.3), with an LVM layout.
lsblk output:
~~~
sda             8:0    0   20G  0 disk 
|-sda1          8:1    0  500M  0 part /boot
`-sda2          8:2    0 19.5G  0 part 
  |-rhel-swap 253:0    0  1.8G  0 lvm  [SWAP]
  `-rhel-root 253:1    0 17.7G  0 lvm  /
sdb             8:16   0   30G  0 disk 
`-sdb1          8:17   0   30G  0 part /data1
sdc             8:32   0   30G  0 disk /data2
sdd             8:48   0   16G  0 disk 
`-sdd1          8:49   0   16G  0 part 
~~~

blkid output (/ and /boot have an XFS ftype=0):
~~~
/dev/sda1: UUID="5d3b08b9-4d06-4ba6-9d71-4498966c68a2" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="000a8a6c-01"
/dev/sda2: UUID="0fsCrY-UUI2-270N-20bX-eQqT-i3VT-Afxix0" TYPE="LVM2_member" PARTUUID="000a8a6c-02"
/dev/sdc: UUID="f9502b09-f530-4e82-a7e9-5602a647e65d" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdd1: UUID="0b235933-4855-4779-b016-2607a51ceb6a" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6a994b3d-01"
/dev/sdb1: LABEL="DATA1" UUID="4e3f5651-455b-4935-85e3-a31693615eb8" BLOCK_SIZE="4096" TYPE="ext3"
/dev/mapper/rhel-swap: UUID="a52132c0-46c8-4959-8f71-a2cd7c1580d7" TYPE="swap"
/dev/mapper/rhel-root: UUID="7b8954b3-ed5e-47d7-a3e8-c8b7b47d9346" BLOCK_SIZE="512" TYPE="xfs"
~~~

Actual results:
~~~
Jan 26 10:11:53 localhost systemd[1]: Starting System Upgrade...
Jan 26 10:11:53 localhost upgrade[663]: starting upgrade hook
Jan 26 10:11:53 localhost upgrade[663]: /bin/upgrade: line 19: /sysroot/var/tmp/system-upgrade.state: Read-only file system
Jan 26 10:11:53 localhost upgrade[663]:   WARNING: locking_type (4) is deprecated, using --sysinit --readonly.
Jan 26 10:11:53 localhost upgrade[663]:   Allowing activation with --readonly --sysinit.
Jan 26 10:11:53 localhost upgrade[663]:   WARNING: PV /dev/sda2 in VG rhel is using an old PV header, modify the VG to update.
Jan 26 10:11:53 localhost upgrade[663]:   2 logical volume(s) in volume group "rhel" now active
Jan 26 10:11:53 localhost upgrade[663]: //lib/dracut/hooks/upgrade/50-do-upgrade.sh: line 17: [: missing `]'
Jan 26 10:11:53 localhost upgrade[663]: Spawning container sysroot on /sysroot.
Jan 26 10:11:53 localhost upgrade[663]: Press ^] three times within 1s to kill container.
Jan 26 10:11:53 localhost kernel: XFS (sda1): Mounting V4 Filesystem
Jan 26 10:11:54 localhost kernel: XFS (sda1): Ending clean mount
Jan 26 10:11:54 localhost kernel: EXT4-fs (sdb1): mounting ext3 file system using the ext4 subsystem
Jan 26 10:11:54 localhost kernel: EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
Jan 26 10:11:54 localhost kernel: XFS (sdc): Mounting V5 Filesystem
Jan 26 10:11:54 localhost kernel: XFS (sdc): Ending clean mount
Jan 26 10:11:54 localhost kernel: EXT4-fs (sdd1): mounted filesystem with ordered data mode. Opts: (null)
Jan 26 10:11:54 localhost upgrade[663]: mount: unknown filesystem type 'objectivefs'
Jan 26 10:12:10 localhost upgrade[663]: ==> Processing phase `InitRamStart`
Jan 26 10:12:10 localhost upgrade[663]: ====> * remove_upgrade_boot_entry
Jan 26 10:12:10 localhost upgrade[663]:         Remove boot entry for Leapp provided initramfs.
Jan 26 10:12:11 localhost upgrade[663]: Process Process-158:
Jan 26 10:12:11 localhost upgrade[663]: Traceback (most recent call last):
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Jan 26 10:12:11 localhost upgrade[663]:     self.run()
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
Jan 26 10:12:11 localhost upgrade[663]:     self._target(*self._args, **self._kwargs)
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
Jan 26 10:12:11 localhost upgrade[663]:     actor_instance.run(*args, **kwargs)
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 335, in run
Jan 26 10:12:11 localhost upgrade[663]:     self.process(*args)
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/removeupgradebootentry/actor.py", line 20, in process
Jan 26 10:12:11 localhost upgrade[663]:     remove_boot_entry()
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/removeupgradebootentry/libraries/removeupgradebootentry.py", line 41, in remove_boot_entry
Jan 26 10:12:11 localhost upgrade[663]:     '/bin/mount', '-a'
Jan 26 10:12:11 localhost upgrade[663]:   File "/usr/lib/python2.7/site-packages/leapp/libraries/stdlib/__init__.py", line 188, in run
Jan 26 10:12:11 localhost upgrade[663]:     result=result
Jan 26 10:12:11 localhost upgrade[663]: CalledProcessError: Command ['/bin/mount', '-a'] failed with exit code 32.
Jan 26 10:12:11 localhost upgrade[663]: ==========================================================================================================
Jan 26 10:12:11 localhost upgrade[663]: Actor remove_upgrade_boot_entry unexpectedly terminated with exit code: 1 - Please check the above details
Jan 26 10:12:11 localhost upgrade[663]: ==========================================================================================================
~~~

Expected results:
Works :)

Additional info:
The workaround below helped to move forward.

After the `leapp ugprade` phase, and before rebooting, please append the following parameter into the "RHEL 8 Upgrade Initramfs" boot entry in grub.cfg (or do it from the grub menu while rebooting):
~~~
rd.break=upgrade
~~~

Once you are on a shell prompt, please run the command below:
~~~
# sed -i 's/locking_type = 4/locking_type = 1/' /etc/lvm/lvm.conf
# lvm vgchange -ay --config ' global {locking_type=1} '
# lvm vgck --updatemetadata rhel
# sed -i 's/locking_type = 1/locking_type = 4/' /etc/lvm/lvm.conf
# exit
~~~

As per `man vgck`:
~~~
       Rewrite VG metadata to correct problems.
       vgck --updatemetadata VG
           [ COMMON_OPTIONS ]
~~~

Comment 2 Petr Stodulka 2021-02-11 10:48:14 UTC

Thank you Chris for the report and reproducer! We will definitely try it to see what we can do here.

Comment 3 Michal Reznik 2021-02-11 14:44:06 UTC

@Chris, could you please share the original "/etc/lvm/lvm.conf" file before the sed command?

Comment 4 Christophe Besson 2021-02-11 15:03:01 UTC

It was needed in my case, or vgck cannot work.

# mkdir /tmp/initrd && cd /tmp/initrd
# /usr/lib/dracut/skipcpio /boot/initramfs-upgrade.x86_64.img | gzip -d | cpio -id
# cat etc/lvm/lvm.conf
global {
locking_type = 4
use_lvmetad = 0
}

It's the one included in the initramfs, not the one from the rootfs.

Comment 12 David Teigland 2022-05-02 20:42:25 UTC

(In reply to Christophe Besson from comment #0)
> Description of problem:
> Customer ran a leapp upgrade and the first step worked fine.
> Once they rebooted on the upgrade initramfs, they got a python traceback,
> leapp being unable to issue a `mount -a`. This is a consequence of a warning
> printed out just before the issue:
> """
> WARNING: PV /dev/sda2 in VG rhel is using an old PV header, modify the VG to
> update.

I'm not seeing the connection between this lvm message and mount -a failing.

> Actual results:
> ~~~
> Jan 26 10:11:53 localhost systemd[1]: Starting System Upgrade...
> Jan 26 10:11:53 localhost upgrade[663]: starting upgrade hook
> Jan 26 10:11:53 localhost upgrade[663]: /bin/upgrade: line 19:
> /sysroot/var/tmp/system-upgrade.state: Read-only file system
> Jan 26 10:11:53 localhost upgrade[663]:   WARNING: locking_type (4) is
> deprecated, using --sysinit --readonly.
> Jan 26 10:11:53 localhost upgrade[663]:   Allowing activation with
> --readonly --sysinit.
> Jan 26 10:11:53 localhost upgrade[663]:   WARNING: PV /dev/sda2 in VG rhel
> is using an old PV header, modify the VG to update.
> Jan 26 10:11:53 localhost upgrade[663]:   2 logical volume(s) in volume
> group "rhel" now active

The "old PV header" message is a warning and does not prevent the activation, which was successful here.

Comment 13 Vinzenz Feenstra [evilissimo] 2022-05-02 20:53:32 UTC

I agree here with David, if the activation would have failed, we wouldn't see a backtrace from leapp saying that 'mount -a' failed. Which reports a failure because of an unknown filesystem type objectivefs, which by all I can tell is a way to mount S3 buckets.

Comment 14 Christophe Besson 2022-05-03 07:36:14 UTC

Thanks for the review, indeed the `mount -a` issue here is caused by the unknown filesystem.
The issue has likely been fixed by removing the fstab entry in the same try than my suggestion to fix these warnings.
And the other cases have been added because these warnings have been seen (in addition to other problems causing the real issue).
So I guess this case can be closed as not a bug.

Comment 15 Vinzenz Feenstra [evilissimo] 2022-05-03 07:49:46 UTC

After consideration that 'objectivefs' indeed is not supported during the upgrade and verifying with the LVM team that the warnings are rather informative and not the root cause of the issue, we'll close this BZ as NOT A BUG.

Thank you anyway for the report.

Comment 16 bryanhoop 2024-06-28 16:54:30 UTC

I just got bit by this during a remote upgrade.

Might it be prudent to scan /etc/fstab for unsupported fs during preupgrade checks and warn the user?

Note You need to log in before you can comment on or make changes to this bug.