Bug 1919989

Summary: ReaR Restore Fails At Disk Recreation Step With Volume Group Messages.
Product: Red Hat Enterprise Linux 7 Reporter: Bernie Hoefer <bhoefer>
Component: rearAssignee: Pavel Cahyna <pcahyna>
Status: CLOSED NOTABUG QA Contact: CS System Management SST QE <rhel-cs-system-management-subsystem-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.9CC: ovasik, pcahyna, sorkim
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-26 09:12:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ReaR log file. none

Description Bernie Hoefer 2021-01-25 14:16:13 UTC 
Created attachment 1750527 [details]
ReaR log file.

Description of problem:
I am using ReaR for the first time, using
[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-relax-and-recover_rear]
...as my guide.  I am not sure if I'm just doing something wrong, but when I try to restore my test machine from the ISO created, it fails at the "disk recreation script" step.


Version-Release number of selected component (if applicable):
rear-2.4-13.el7.x86_64


Steps to Reproduce:
1. Create RHEL 7.9 virtual machine using a minimal install.
2. Post-installation, add a 2nd disk to the virtual machine.  Create 2
   partitions; add the 1st partition as a physical volume to the volume
   group.  Extended the root logical volume and file system to use this
   new space.  Make the 2nd partition on the new drive its own file system.
   Final layout of disks should look like this:
     # lsblk
     NAME                       MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     sr0                         11:0    1 1024M  0 rom  
     vda                        252:0    0   10G  0 disk 
     ├─vda1                     252:1    0  512M  0 part /boot
     └─vda2                     252:2    0  9.5G  0 part 
       ├─rhel_rhel79--gold-root 253:0    0  9.5G  0 lvm  /
       └─rhel_rhel79--gold-swap 253:1    0  508M  0 lvm  [SWAP]
     vdb                        252:16   0    1G  0 disk 
     ├─vdb1                     252:17   0  500M  0 part 
     │ └─rhel_rhel79--gold-root 253:0    0  9.5G  0 lvm  /
     └─vdb2                     252:18   0  523M  0 part /opt/3rdparty
3. Create a ReaR ISO per chapter 27 of the RHEL 7
   _System Administrator's Guide_.
4. Boot the virtual machine off of RHEL ISO; drop to shell and blank both
   disks by using:
     # dd if=/dev/zero of=/dev/vda bs=1000 count=1000
     1000+0 records in
     1000+0 records out
     1000000 bytes (1.0 MB) copied, 0.0475139 s, 21.0 MB/s
     # dd if=/dev/zero of=/dev/vdb bs=1000 count=1000
     1000+0 records in
     1000+0 records out
     1000000 bytes (1.0 MB) copied, 0.0548423 s, 18.2 MB/s
5. Try to restore the machine using the ISO created in step #3 and the
   instructions in chapter 27 of the RHEL 7 _System Administrator's Guide_.


Actual results:
Restore fails at this step:
  User confirmed disk recreation script
  Start system layout restoration.
  Creating partitions for disk /dev/vda (msdos)
  Creating partitions for disk /dev/vdb (msdos)
  Creating LVM PV /dev/vda2
  Creating LVM PV /dev/vdb1
  Creating LVM VG 'rhel_rhel79-gold'; Warning: some properties may not be preserved...
  The disk layout recreation script failed


Expected results:
Successful restoration.


Additional info:
Looking at the log file (see attached), the disk recreation seems to fail at line 528:
  +++ lvm vgcreate --physicalextentsize 4096k rhel_rhel79-gold /dev/vda2 /dev/vdb1
    WARNING: Failed to connect to lvmetad. Falling back to device scanning.
    A volume group called rhel_rhel79-gold already exists.
Does the volume group already get created at log file lines 455 and 489 with the "lvm vgchange -a n" commands?  The output of those commands have some warnings about inconsistent metadata, but I do not know why that would be.

I took a snapshot of my virtual machine before creating the ReaR ISO and wiping the disks, so I can always go back to it for BZ troubleshooting purposes.
Comment 2 Pavel Cahyna 2021-01-25 14:49:39 UTC 
Are you sure the dd commands are sufficient to erase any traces of the previously existing PVs/VGs ? (Also, I would use a multiple of 512 as the block size, although that's probably not the problem.)

I suggest to wipe the disks completely, or use freshly created disk images, to rule out the possibility that there are some traces of the previous LVM setup lingering on the disks.
Comment 3 Bernie Hoefer 2021-01-25 15:26:58 UTC 
(In reply to Pavel Cahyna from comment #2)
===
> Are you sure the dd commands are sufficient to erase any traces of the
> previously existing PVs/VGs ?
===

That was apparently the problem.  I just tried it again, this time wiping the partitions 1st and then the disks:

  dd if=/dev/zero of=/dev/vdb2 bs=1024 count=1024
  dd if=/dev/zero of=/dev/vdb1 bs=1024 count=1024
  dd if=/dev/zero of=/dev/vdb bs=1024 count=1024
  dd if=/dev/zero of=/dev/vda2 bs=1024 count=1024
  dd if=/dev/zero of=/dev/vda1 bs=1024 count=1024
  dd if=/dev/zero of=/dev/vda bs=1024 count=1024

Whenever I've wanted to `blow away` a machine to install something else, zeroing-out just the disk was good enough since that removes its partition table.  But I suppose with ReaR re-creating the partition table, *exactly*, it shouldn't be a surprise that it then sees what was left on the disk where those partitions existed/are recreated.

Like I wrote in this Bugzilla ticket's description, this is my first exploration of ReaR.  I guess it is an uncommon use-case to do a restore on the same disks from which the ReaR backup was made.

Thanks again for your help.
Comment 4 Pavel Cahyna 2021-01-26 09:12:05 UTC 
(In reply to Bernie Hoefer from comment #3)

> Whenever I've wanted to `blow away` a machine to install something else,
> zeroing-out just the disk was good enough since that removes its partition
> table.  But I suppose with ReaR re-creating the partition table, *exactly*,
> it shouldn't be a surprise that it then sees what was left on the disk where
> those partitions existed/are recreated.

Indeed, and I believe that modern fdisk even leaves about 1 MB between the partition table and the first partition, in which case wiping less than 1 MB is certainly not enough.

> Like I wrote in this Bugzilla ticket's description, this is my first
> exploration of ReaR.  I guess it is an uncommon use-case to do a restore on
> the same disks from which the ReaR backup was made.
> 
> Thanks again for your help.

Not sure if it is an uncommon use-case, probably not that uncommon, but anyway ReaR does not seem to have any built-in automation for wiping the previous disk content. Given the destructiveness of such operation and the potential of going wrong, it may be actually a good thing to not have it.

You are welcome and good luck with ReaR. Make sure to test your backups before use, as you just did. Whole system backup and restore is a complex and fragile process, many things can go wrong.
Comment 5 Pavel Cahyna 2021-03-17 11:38:33 UTC 
By the way, there is a useful command for this, you don't need to use dd manually. It is called wipefs.
First call it on partitions
wipefs -a /dev/vdb2
...
and then on entire disks
wipefs -a /dev/vdb
...

There is now code upstream to perform this, and another bugs open for this RFE, so it will probably get fixed in the future (at least when we do a rebase to a new version). bz1925530 and bz1925531