Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2006946

Summary: [RFE] need a way to faithfully restore volume groups to clean disks
Product: Red Hat Enterprise Linux 8 Reporter: Pavel Cahyna <pcahyna>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
lvm2 sub component: Command-line tools QA Contact: cluster-qe <cluster-qe>
Status: CLOSED DEFERRED Docs Contact:
Severity: unspecified    
Priority: unspecified CC: agk, heinzm, jbrassow, msnitzer, prajnoha, teigland, thornber, zkabelac
Version: 8.4Keywords: FutureFeature
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2037415 (view as bug list) Environment:
Last Closed: 2022-01-05 15:48:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2037415    

Description Pavel Cahyna 2021-09-22 17:17:30 UTC
Background:

I maintain a disaster recovery tool called Relax-and-Recover (ReaR). The goal of the tool is to save a representation of the current state of a system in order to be able to recreate it on new hardware in case of a disaster. In particular, this includes the storage layout (partitions, LVM, fileystems, RAIDs). The tool keeps a serialized representation of the storage layout and is able to recreate the layout that is then populated by a backup/restore tool (like tar or rsync or something more sophisticated).

For LVM, a faithful restoration of the layout is achieved by saving it using vgcfgbackup and restoring it using vgcfgrestore. An important limitation of this method is the inability of proper restoration of any logical volumes that have kernel metadata. The fundamental reason is that vgcfgrestore is not really designed for restoration to clean disks: the intent is to restore the LVM metadata on disks which already have the kernel metadata on them. As a result, ReaR is able to properly restore only volume groups that do not contain anything but striped and linear logical volumes (these two layouts do not use kernel metadata). If the volume groups contain anything else, esp. thin pools, ReaR is not very reliable and has to resort to hacks - see bz1747468.

There is another method implemented in ReaR, called MIGRATION_MODE. It avoids vgcfgrestore and creates the VG and its LVs using the vgcreate and lvcreate commands. This is the supported way to recreate the LVM layout, problem is, it is not guaranteed to restore all the attributes of LVs (see https://github.com/rear/rear/issues/2222#issuecomment-527151886), because the tool has to inspect the output of tools like lvs and construct the command line of lvcreate. This is not a maintainable approach, because even if support of all the attributes is added to ReaR, new attributes ate being added to LVM all the time, so ReaR will be perpetually behind.

Description of problem:

For backup and restore purposes, we need to be able to save the LVM layout (into a file or a set of files) and then restore on a clean set of disks. There is currently no tool that can do that (LVM has rich reporting capabilities, but does not have anything that is able to recreate VGs and LVs using the reports that it has produced). The closest to what we need is the the existing vgcfgrestore tool, so extending it seems like the most straightforward approach. The main problem with the tool are logical volumes that use kernel metadata. If the tool restores them on a disk that previously has not had them, the volumes will not be in a consistent state. Worse, there is a risk of misbehavior if the disks contain seemingly valid kernel metadata that are accidentally found at the place where vgcfgrestore has recreated the volumes.

The right approach is probably adding some functionality to vgcfgrestore that will allow it to restore volumes without activating them and clear and reinitialize kernel metadata on volumes that are using them. The resulting layout will then have thin pools/volumes, raids, caches with all the correct attributes, but empty. One then can proceed to creating filesystems on them and filling them with data.

For some volume types like snapshots this approach does not make sense, so they will have to be omitted from the resulting layout. This could be generalized as a capability to restore only a selected subset of LVs instead of the whole VG.

Note: something similar will be needed also for MDRAID.

Comment 2 David Teigland 2021-09-22 18:30:05 UTC
There are a number of cases that require something similar.  There are three basic parts of a solution:

1. A format/syntax to describe an lvm layout (or storage more broadly.)

2. A way to generate this description from looking at the existing system state.

3. A way to create the lvm/storage layout from the description.

For lvm, and I'm guessing others, 3 would involve running a series of ordinary create/change/convert commands, and not trying to use metadata backup restore.

There are some efforts under way on parts of this which should be connected to this RFE.

Comment 3 Zdenek Kabelac 2021-09-29 15:36:02 UTC
We have original discussion with Pavel - main idea is enhancing 'vgcfgrestore' with options not just to 'restore' lvm2 metadata and the drive where original lives on to also be usable on devices which are 'completely' new.

ReaR is used for restoring storage. User may want to 'restore' setup on the same sized set of 'drives' or possibly on similarly 'sized' set of devices, but with different layout.

This normally works for 'metadata-less' devices like  'linears/striped' - but as soon as LV uses some sort of metadata - other tools are 'lost' - and usually try to badly reimplement all the 'lvm2' wisdom.

So we can provide several relatively 'simple' to add extension to our existing tooling.  So we can provide a more simple mechanism to restore i.e.  'thin-pool  with empty 10 thin LVs'  with matching names & uuid.

Or having some layout for cached/mirrored LV.

As for comment 2 - current 'ReaR' tool approach was partially going in direction of 'recreating'   LVs - however this is technically very hard - and actually lvm2 even doesn't provide mechanism how to simply copy all 'attributes' of LV.
Adding mechanism into lvm2 codebase to 'setup' LV metadata into 'wanted' state is more approachable solution - since it keeps the  lvm2 knowledge within out codebase - and ReaR can use 'high-level' abstraction without require to implement lvm2 knowledge in this tool.

So in our discussion with Pavel - I've been having some question - what is the meaning of 'snapshots' during such restore - as clearly these are things which do not have much 'logical' sense behind restoring them - if there are no 'real' data behind it.
Also 'random' caching may be also a questionable task.

So ATM we may need 2 extending options (ATM with just 'randomly picked descriptive names):

--initilize-metadada  - ensuring i.e. thin-pool knows about thinLVs
--prune-useless-lvs   - dropping snapshots/ caches ?

Lvm2 should be capable to 'transfer' set LVs even on different PVs - but this might be currently 'too hard' (involving actually going  1-by-1  creation step and transferring attributes )
- so the focus should go on restoring on same sized devices - which relates to existing customer cases.

Comment 4 Pavel Cahyna 2021-10-04 18:12:46 UTC
Hello @zkabelac , thanks for the review. I have two questions:

> --prune-useless-lvs   - dropping snapshots/ caches ?

I understand snapshots, but why are caches a problem? I think it is perfectly valid to recreate caches with the same attributes but empty (with no cached data in them).

> Lvm2 should be capable to 'transfer' set LVs even on different PVs - but
> this might be currently 'too hard' (involving actually going  1-by-1 
> creation step and transferring attributes )
> - so the focus should go on restoring on same sized devices - which relates
> to existing customer cases.

Would it be hard to extend it to same sized or bigger devices? (I understand that restoring to smaller devices would be difficult.)