2117937 – rear recover failing

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2117937 - rear recover failing

Summary: rear recover failing

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	rear
Sub Component:
Version:	9.0
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Pavel Cahyna
QA Contact:	David Jež
Docs Contact:
URL:
Whiteboard:
Depends On:	2119473 2121237
Blocks:
TreeView+	depends on / blocked

Reported:	2022-08-12 18:46 UTC by eric_hagen
Modified:	2022-11-15 11:48 UTC (History)
CC List:	3 users (show)
Fixed In Version:	rear-2.6-15.el9
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-11-15 10:42:27 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-131040	0	None	None	None	2022-08-12 18:49:59 UTC
Red Hat Product Errata	RHBA-2022:8246	0	None	None	None	2022-11-15 10:42:43 UTC

Description eric_hagen 2022-08-12 18:46:39 UTC

Description of problem:
rear recover failing - PVID which is already used for missing device

Version-Release number of selected component (if applicable):
rear-2.6-11.el9.x86_64

How reproducible:
VM using Secure Boot, to ISO 

Steps to Reproduce:
rear -d -v mkrescue 
Finishes, produces boot ISO, and backup.
#/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (20182) - No such process
#Exit task '(( EXIT_FAIL_MESSAGE )) && echo 'rear mkrescue failed, check /var/log/rear

Actual results:
rear -D -v recover
# Error:
# rear recover LAYOUT_CODE_RUN PVID which is already used for missing device

UserInput -I LAYOUT_FILE_CONFIRMATION needed in 
/usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26

Disk '/dev/sda': creating 'gpt' partition table
Disk '/dev/sda': creating partition number 1 with name ''EFI System Partition''
Disk '/dev/sda': creating partition number 2 with name ''sda2''
Disk '/dev/sda': creating partition number 3 with name ''sda3''
Disk '/dev/sda': creating partition number 4 with name ''sda4''
Creating LVM PV /dev/sda3
UserInput -I LAYOUT_CODE_RUN needed in 
/usr/share/rear/layout/recreate/default/200_run_layout_code.sh line 127

++ local disk_size=522003415040
++ [[ -n 522003415040 ]]
++ BugIfError 'Could not determine size of disk /dev/sda, please file a bug.'
++ ((  0 != 0  ))
++ [[ 522003415040 -gt 0 ]]

2022-07-20 19:36:38.504048816 Creating LVM PV /dev/sda3
+++ Print 'Creating LVM PV /dev/sda3'
+++ lvm vgchange -a n emss
  Volume group "emss" not found
  Cannot process volume group emss
+++ true
+++ lvm pvcreate -ff --yes -v --uuid Idoq6T-48Kr-rrHB-A2vh-iNUp-7g6V-q1XCTr --norestorefile /dev/sda3
  Wiping signatures on new PV /dev/sda3.
  WARNING: adding device /dev/sda3 with PVID Idoq6T48KrrrHBA2vhiNUp7g6Vq1XCTr which is already used for missing device.
/var/lib/rear/layout/diskrestore.sh: line 131:  5824 Segmentation fault      (core dumped) lvm pvcreate -ff --yes -v --uuid "Idoq6T-48Kr-rrHB-A2vh-iNUp-7g6V-q1XCTr" --norestorefile /dev/sda3 1>&2
## retry
+++ Print 'Creating LVM PV /dev/sda3'
+++ lvm vgchange -a n emss
  Volume group "emss" not found
  Cannot process volume group emss
+++ true
+++ lvm pvcreate -ff --yes -v --uuid Idoq6T-48Kr-rrHB-A2vh-iNUp-7g6V-q1XCTr --norestorefile /dev/sda3
  Wiping signatures on new PV /dev/sda3.
  WARNING: adding device /dev/sda3 with PVID Idoq6T48KrrrHBA2vhiNUp7g6Vq1XCTr which is already used for missing device.
/var/lib/rear/layout/diskrestore.sh: line 131:  5860 Segmentation fault      (core dumped) lvm pvcreate -ff --yes -v --uuid "Idoq6T-48Kr-rrHB-A2vh-iNUp-7g6V-q1XCTr" --norestorefile /dev/sda3 1>&2
++ ((  139 == 0  ))


Expected results:
ReaR creates bootable system.

Additional info:

# system built sith the following anaconda commands.

ignoredisk --only-use=sda
# Clear the Master Boot Record
zerombr
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information
part /boot --fstype="xfs" --ondisk=sda --size=1024
part pv.127 --fstype="lvmpv" --ondisk=sda --size=229332
part /boot/efi --fstype="efi" --ondisk=sda --size=600 --fsoptions="defaults,uid=0,gid=0,umask=077,shortname=winnt"
volgroup emss --pesize=4096 pv.127
logvol /var/tmp --fstype="xfs" --size=4096 --name=var_tmp --vgname=emss
logvol /home --fstype="xfs" --size=102400 --name=home --vgname=emss
logvol swap --fstype="swap" --size=4042 --name=swap --vgname=emss
logvol /var/log/audit --fstype="xfs" --size=11264 --name=var_log_audit --vgname=emss
logvol /var/log --fstype="xfs" --size=11264 --name=var_log --vgname=emss
logvol /var --fstype="xfs" --size=20480 --name=var --vgname=emss
logvol /tmp --fstype="xfs" --size=4096 --name=tmp --vgname=emss
logvol / --fstype="xfs" --grow --size=1024 --name=root --vgname=emss

# Manually Extended 
fdisk /dev/sda
n
p
t
8e
p
w

pvcreate /dev/sda4
vgextend emss /dev/sda4






# Note 
lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME                               KNAME     PKNAME    TRAN   TYPE FSTYPE       SIZE MOUNTPOINT
/dev/sda                           /dev/sda                   disk              300G 
|-/dev/sda1                        /dev/sda1 /dev/sda         part vfat         600M /boot/efi
|-/dev/sda2                        /dev/sda2 /dev/sda         part xfs            1G /boot
|-/dev/sda3                        /dev/sda3 /dev/sda         part LVM2_member  224G 
| |-/dev/mapper/emss-root          /dev/dm-0 /dev/sda3        lvm  xfs           70G /
| |-/dev/mapper/emss-swap          /dev/dm-1 /dev/sda3        lvm  swap         3.9G [SWAP]
| |-/dev/mapper/emss-tmp           /dev/dm-2 /dev/sda3        lvm  xfs            4G /tmp
| |-/dev/mapper/emss-var_log       /dev/dm-3 /dev/sda3        lvm  xfs           11G /var/log
| |-/dev/mapper/emss-var_log_audit /dev/dm-4 /dev/sda3        lvm  xfs           11G /var/log/audit
| |-/dev/mapper/emss-var           /dev/dm-5 /dev/sda3        lvm  xfs           20G /var
| |-/dev/mapper/emss-home          /dev/dm-6 /dev/sda3        lvm  xfs          100G /home
| `-/dev/mapper/emss-var_tmp       /dev/dm-7 /dev/sda3        lvm  xfs            4G /var/tmp
`-/dev/sda4                        /dev/sda4 /dev/sda         part LVM2_member 74.5G 

pvscan -v
  PV /dev/sda3   VG emss            lvm2 [223.95 GiB / 0    free]
  PV /dev/sda4   VG emss            lvm2 [74.45 GiB / 74.45 GiB free]
  Total: 2 [<298.41 GiB] / in use: 2 [<298.41 GiB] / in no VG: 0 [0   ]

vgscan
  Found volume group "emss" using metadata type lvm2

parted /dev/sda print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 322GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  630MB   629MB   fat32        EFI System Partition  boot, esp
 2      630MB   1704MB  1074MB  xfs
 3      1704MB  242GB   240GB                                      lvm
 4      242GB   322GB   79.9GB


/root/tmp/rear.gqM3yWE9PkZfm8M/tmp/
parted
BYT;
/dev/sda:322GB:scsi:512:512:gpt:VMware Virtual disk:;
1:1049kB:630MB:629MB:fat32:EFI System Partition:boot, esp;
2:630MB:1704MB:1074MB:xfs::;
3:1704MB:242GB:240GB:::lvm;
4:242GB:322GB:79.9GB:::;

partitions-data 
1 629145600 1048576
2 1073741824 630194176
3 240472031232 1703936000
4 79946563072 242175967232

partitions
1 629145600 1048576 EFI%20System%20Partition boot,esp
2 1073741824 630194176 rear-noname none
3 240472031232 1703936000 rear-noname lvm
4 79946563072 242175967232 rear-noname none



## output of: 
rear -d -v mkrescue 
Relax-and-Recover 2.6 / 2020-06-17
Running rear mkrescue (PID 29677)
Using log file: /var/log/rear/rear-rhel9.log
Running workflow mkrescue on the normal/original system
Found EFI system partition /dev/sda1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using autodetected kernel '/boot/vmlinuz-5.14.0-70.17.1.el9_0.x86_64' as kernel in the recovery system
Creating disk layout
Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf
Using guessed bootloader 'EFI' (found in first bytes on /dev/sda)
Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
Creating recovery system root filesystem skeleton layout
Handling network interface 'ens192'
ens192 is a physical device
Handled network interface 'ens192'
Cannot include default keyboard mapping (no 'defkeymap.*' found in /lib/kbd/keymaps)
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/redhat/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-rhel9.log into initramfs as '/tmp/rear-rhel9-partial-2022-07-21T19:43:25+00:00.log'
Copying files and directories
Copying binaries and libraries
Copying all kernel modules in /lib/modules/5.14.0-70.17.1.el9_0.x86_64 (MODULES contains 'all_modules')
Copying all files in /lib*/firmware/
Skip copying broken symlink '/etc/mtab' target '/proc/38702/mounts' on /proc/ /sys/ /dev/ or /run/
Broken symlink '/usr/lib/modules/5.14.0-70.17.1.el9_0.x86_64/source' in recovery system because 'readlink' cannot determine its link target
Broken symlink '/usr/lib/modules/5.14.0-70.17.1.el9_0.x86_64/build' in recovery system because 'readlink' cannot determine its link target
Testing that the recovery system in /root/tmp/rear.gqM3yWE9PkZfm8M/rootfs contains a usable system
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (423419609 bytes) in 21 seconds
GRUB2 modules to load: fat part_gpt xfs
Did not find /boot/grub2/locale files (minor issue for UEFI ISO boot)
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-rhel9.iso (463M)
Copying resulting files to nfs location
Saving /var/log/rear/rear-rhel9.log as rear-rhel9.log to nfs location
Copying result files '/var/lib/rear/output/rear-rhel9.iso /root/tmp/rear.gqM3yWE9PkZfm8M/tmp/VERSION /root/tmp/rear.gqM3yWE9PkZfm8M/tmp/README /root/tmp/rear.gqM3yWE9PkZfm8M/tmp/rear-rhel9.log' to /root/tmp/rear.gqM3yWE9PkZfm8M/outputfs/rhel9 at nfs location
Exiting rear mkrescue (PID 29677) and its descendant processes ...
Running exit tasks
You should also rm -Rf --one-file-system /root/tmp/rear.gqM3yWE9PkZfm8M

Attached mkrescue-rear-rhel9.log

Comment 1 Pavel Cahyna 2022-08-15 08:53:03 UTC

Thank you for the detailed report. "lvm pvcreate" segfaults, can you please collect a core dump? You need to set this before running the problematic command:

sysctl -w kernel.core_pattern=core.%e.%p

because by default core dumps are not saved.

Comment 2 Pavel Cahyna 2022-08-17 15:55:17 UTC

I suspect that the segfault is related to the message
WARNING: adding device /dev/sda3 with PVID Idoq6T48KrrrHBA2vhiNUp7g6Vq1XCTr which is already used for missing device.

I have not seen this with successful restores.

Comment 3 Lukáš Zaoral 2022-08-18 08:35:12 UTC

This is a bug in lvm2 unrelated to ReaR itself.  Reproducer:

1. Get an empty disk (e.g. /dev/vdb).
2. Install lvm2 package.
2. Create one partition on /dev/vdb (e.g. using fdisk).
3. # pvcreate /dev/vdb1
4. # pvscan -u
  PV /dev/vdb1 with UUID LQbrRY-PusD-1f3b-bW0U-akcG-wEwy-s2DqBA                       lvm2 [<10.00 GiB]
  Total: 1 [<10.00 GiB] / in use: 0 [0   ] / in no VG: 1 [<10.00 GiB]
5. # wipefs -fa /dev/vdb1
6. # lvm pvcreate -ff --yes -v --uuid LQbrRY-PusD-1f3b-bW0U-akcG-wEwy-s2DqBA --norestorefile /dev/vdb1
  Devices file PVID LQbrRYPusD1f3bbW0UakcGwEwys2DqBA not found on device /dev/vdb1.
  Wiping signatures on new PV /dev/vdb1.
  WARNING: adding device /dev/vdb1 with PVID LQbrRYPusD1f3bbW0UakcGwEwys2DqBA which is already used for missing device.
Segmentation fault (core dumped)

Coredump stack trace:

#0  0x00007f65a449f8e2 __strcmp_evex (libc.so.6 + 0xcd8e2)
#1  0x000055e3471c76ec device_id_add (lvm + 0xb96ec)
#2  0x000055e3471914fa pvcreate_each_device (lvm + 0x834fa)
#3  0x000055e34717715d pvcreate (lvm + 0x6915d)
#4  0x000055e34716a935 lvm_run_command (lvm + 0x5c935)
#5  0x000055e34716c561 lvm2_main (lvm + 0x5e561)
#6  0x00007f65a4411eb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
#7  0x00007f65a4411f60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
#8  0x000055e347146e45 _start (lvm + 0x38e45)
ELF object binary architecture: AMD x86-64

Comment 4 Pavel Cahyna 2022-08-18 10:16:36 UTC

Please try to recover again after moving /etc/lvm/devices in the rescue system away: mv /etc/lvm/devices /etc/lvm/devices.orig.

For new backups, I would recommend omitting the directory from the rescue system entirely, which should revert LVM tools to the previous behavior:
add
COPY_AS_IS_EXCLUDE+=( /etc/lvm/devices )
to /etc/rear/local.conf.

Comment 5 eric_hagen 2022-08-18 20:33:39 UTC

Confirmed work around for restore worked by moving /etc/lvm/devices.
Confirmed work around for backup / restore addtion to /etc/rear/local.conf of:
COPY_AS_IS_EXCLUDE+=( /etc/lvm/devices )
Restore completed without crashing. 
Thank you very much for your work on this.

Comment 13 errata-xmlrpc 2022-11-15 10:42:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rear bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8246

Note You need to log in before you can comment on or make changes to this bug.