RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1747468 - Unable to restore VG with Thin volumes: "Thin pool rhel-pool00-tpool (XXX:X) transaction_id is X, while expected Y"
Summary: Unable to restore VG with Thin volumes: "Thin pool rhel-pool00-tpool (XXX:X) ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: rear
Version: 8.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: Pavel Cahyna
QA Contact: David Jež
Prerana Sharma
URL:
Whiteboard:
Depends On:
Blocks: 2004178
TreeView+ depends on / blocked
 
Reported: 2019-08-30 14:02 UTC by Renaud Métrich
Modified: 2024-10-01 16:19 UTC (History)
8 users (show)

Fixed In Version: rear-2.6-3.el8
Doc Type: Enhancement
Doc Text:
.Errors when restoring LVM with thin pools do not happen anymore With this enhancement, ReaR now detects when thin pools and other logical volume types with kernel metadata (for example, RAIDs and caches) are used in a volume group (VG) and switches to a mode where it recreates all the logical volumes (LVs) in the VG using lvcreate commands. Therefore, LVM with thin pools are restored without any errors. NOTE: This new method does not preserve all the LV properties, for example LVM UUIDs. A restore from the backup should be tested before using ReaR in a Production environment in order to determine whether the recreated storage layout matches the requirements.
Clone Of:
: 2004178 (view as bug list)
Environment:
Last Closed: 2021-11-09 18:53:41 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3031921 0 None None None 2019-09-09 11:58:29 UTC
Red Hat Product Errata RHEA-2021:4344 0 None None None 2021-11-09 18:53:57 UTC

Description Renaud Métrich 2019-08-30 14:02:28 UTC 
Description of problem:

The code in PR https://github.com/rear/rear/pull/1806 (BZ #1450667) added support for restoring Thin volumes into ReaR 2.4.
Unfortunately it appears it doesn't work as planned.

1. When restoring the exact same system (which already had the VG containing the thin volumes configured at the time of the restore), the code used to delete the Thin pool after using vgcfgrestore is failing due to line 111 not returning an error, making line 112 return an error now

usr/share/rear/layout/prepare/GNU/Linux/110_include_lvm_code.sh:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
105     lvm lvs --noheadings -o lv_name,vg_name,lv_layout | while read lv vg layout ; do
106         # Consider LVs for our VG only
107         [ \$vg == "$vg" ] || continue
108         # Consider Thin Pools only
109         [[ ,\$layout, == *,thin,* ]] && [[ ,\$layout, == *,pool,* ]] || continue
110         # Remove twice, the first time it fails because of thin volumes in the pool
111         lvm lvremove -q -f -y \$vg/\$lv || true    <-- HERE
112         lvm lvremove -q -f -y \$vg/\$lv
113     done
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

At the time PR1806 it was not the case, so there must have been some changed in LVM ...

2. When restoring a new system with same disk capacity (hence with clean new disk, this can be emulated by using "pvremove --force --force /dev/sdaX"), "vgcfgrestore" run but then "lvm lvremove" (line 111) fails with an unrecoverable error

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
+++ lvm lvremove -q -f -y rhel/pool00
  Thin pool rhel-pool00-tpool (253:2) transaction_id is 10, while expected 6.
  Failed to update pool rhel/pool00.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------


Version-Release number of selected component (if applicable):

rear-2.4 including Upstream ReaR


How reproducible:

Always


Steps to Reproduce:
1. Install a system with LVM Thin Provisioning and multiple volumes, keep the swap as a standard LVM volume

  /: thin
  /var: thin
  /home: thin
  /tmp: thin
  swap: linear

2. Create a rescue ISO
3. Recover the system

Actual results:

See above


Expected results:


Additional info:

It looks like to me "vgcfgrestore" cannot be used at all as soon as there is a Thin volume.
I'm pretty sure it was working at the time PR1806 was submitted (June 2018), I need to test this again.

For the demo, I've added a linear swap device because additionally, in scenario 2 ("When restoring a new system"), the swap volume gets restored by vgcfgrestore but doesn't get activated, causing an error when formatting the swap (/dev/vg/swap is not present ...), this is yet an additional issue...
Comment 1 Renaud Métrich 2019-08-30 14:40:41 UTC 
The swap issue seems to happen in scenario 1 as well, due to vgcfgrestore not having enabled the VG it created.

+++ mkswap -U 487d9f6f-93cc-4c6d-be48-a65d21ab1af8 /dev/mapper/rhel-lvswap
/dev/mapper/rhel-lvswap: No such file or directory
Comment 2 Renaud Métrich 2019-08-30 14:47:14 UTC 
For swap issue, a vgchange is missing on line 107:

Upstream Code /usr/share/rear/layout/prepare/GNU/Linux/110_include_lvm_code.sh:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
 95 if lvm vgcfgrestore -f "$VAR_DIR/layout/lvm/${vg}.cfg" $vg >&2 ; then
 96     lvm vgchange --available y $vg >&2
 97 
 98     LogPrint "Sleeping 3 seconds to let udev or systemd-udevd create their devices..."
 99     sleep 3 >&2
100     create_volume_group=0
101     create_logical_volumes=0
102 
103 #
104 # It failed ... restore layout using 'vgcfgrestore --force', but then remove Thin volumes, they are b    roken
105 #
106 elif lvm vgcfgrestore --force -f "$VAR_DIR/layout/lvm/${vg}.cfg" $vg >&2 ; then
107 
108     lvm lvs --noheadings -o lv_name,vg_name,lv_layout | while read lv vg layout ; do
...
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Indeed, even though vgcfgrestore failed, some volumes have been created and can be safely activated.


Apparently RHEL7.5 GA also failed:

+++ lvm lvremove -q -f -y rhel/pool00
  Thin pool rhel-pool00-tpool (252:3) transaction_id is 6, while expected 4.
  Failed to update pool rhel/pool00.

# lvm version
  LVM version:     2.02.177(2)-RHEL7 (2018-01-22)
  Library version: 1.02.146-RHEL7 (2018-01-22)
  Driver version:  4.37.0

So it looks like it never worked properly ...
Comment 3 Zdenek Kabelac 2019-09-02 13:14:27 UTC 
Lvm2 (vgcfgrestore) can't be used this way to restore thin-volumes.

Main purpose of vgcfgrestore (with mandatory --force option) is to let users fix existing thin-pool.

The only proposal for recreating thinLVs on a newly created thin-pool is to call individual 
lvcreate commands so kernel thin-pool metadata can have the information about volumes as well.

So if this is meant to be used for 'data recovery' - the tool recovering LVs need to create
them with proper sizes.
Comment 4 Pavel Cahyna 2019-09-02 13:15:21 UTC 
Would it be feasible to avoid using vgcfgrestore and do the same thing as in migration mode always (or at least always with thin pools)? Properties of segments could be preserved by using lvextend, as in bz1732328#c24.
Comment 5 Pavel Cahyna 2019-09-02 13:16:20 UTC 
zkabelac, I think it is a bug (in ReaR).
Comment 6 Renaud Métrich 2019-09-02 13:17:46 UTC 
Thanks Zdenek for looking into this.
I will be pushing your feedback to ReaR developers so that we come with a proper solution.
Comment 7 Pavel Cahyna 2019-09-02 13:19:00 UTC 
By the way, I think the idea was not to restore thin-volumes this way (using vgcfgrestore), but to restore all other volumes (than thin ones), delete the broken thin ones, and recreate them using lvcreate. This is apparently beacuse vgcfgrestore does not have a way to restore the non-thin volumes only.
Comment 8 Zdenek Kabelac 2019-09-02 13:20:33 UTC 
I'm not familiar with usage of 'rear' tool -  if it's only 'backup' tool for device content - it might be out-of-scope for this tool to be able recreate same sort of storage layout  -

i.e. imagine thinLV on thin-pool this has  cached  raid6 thin-pool data &  raid1 thin-pool metadata - who would be supposed to restore such device stack ?


Anyway it might be possible RFE on  'rear' - but it probably needs different title.
Comment 9 Renaud Métrich 2019-09-02 13:30:09 UTC 
Reopening since component is ReaR, not LVM2, and ReaR needs a fix.

Opened Issue https://github.com/rear/rear/issues/2222 for that Upstream.
Comment 10 Pavel Cahyna 2019-09-02 13:36:21 UTC 
(In reply to Zdenek Kabelac from comment #8)
> I'm not familiar with usage of 'rear' tool -  if it's only 'backup' tool for
> device content - it might be out-of-scope for this tool to be able recreate
> same sort of storage layout  -
> 
> i.e. imagine thinLV on thin-pool this has  cached  raid6 thin-pool data & 
> raid1 thin-pool metadata - who would be supposed to restore such device
> stack ?

It is a recovery tool, so its main responsibility is to backup and recreate the layout, not the content.
Comment 12 Zdenek Kabelac 2019-09-04 11:44:14 UTC 
Deletion of thin-pool works as designed with 'double --force' even for damaged thin-pools.
The problem is 'restoring' same thin-pool with matching  thinID & transactionID.

We we likely need to make some 'new lvm2' command to somehow support recreation of all LVs from scratch.
(as vgcfgbackup/restore relies on the proper disk content)
Comment 35 errata-xmlrpc 2021-11-09 18:53:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rear bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:4344
Note You need to log in before you can comment on or make changes to this bug.