Bug 1021728 - remove issues after pool meta corruption and restoration
remove issues after pool meta corruption and restoration
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.5
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Zdenek Kabelac
cluster-qe@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-21 18:48 EDT by Corey Marthaler
Modified: 2015-07-22 03:37 EDT (History)
10 users (show)

See Also:
Fixed In Version: lvm2-2.02.117-1.el6
Doc Type: Bug Fix
Doc Text:
If the thin-pool was damaged and not repaired, the lvremove failed to remove such pool, as it required to activated such volume. If the user has no intention to preserve such broken pool, the tool now supports double --force --force option to proceed with removal of such pool volume.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-07-22 03:37:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2013-10-21 18:48:01 EDT
Description of problem:
I have yet to successfully remove thin snap or thin origin volumes after corrupting and swapping pool meta devices.


SCENARIO - [swap_deactive_thin_pool_meta_device_w_linear]
Swap a _tmeta device with newly created linear LV while pool is deactivated
Making origin volume
lvcreate --thinpool POOL --zero n -L 1G snapper_thinp
Sanity checking pool device metadata
(thin_check /dev/mapper/snapper_thinp-POOL_tmeta)
examining superblock
examining devices tree
examining mapping tree
lvcreate --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate -V 1G -T snapper_thinp/POOL -n other1
lvcreate --virtualsize 1G -T snapper_thinp/POOL -n other2
lvcreate -V 1G -T snapper_thinp/POOL -n other3
lvcreate --virtualsize 1G -T snapper_thinp/POOL -n other4
lvcreate -V 1G -T snapper_thinp/POOL -n other5
Making snapshot of origin volume
lvcreate -K -s /dev/snapper_thinp/origin -n snap
Create new device to swap in as the new _tmeta device
Dumping current pool metadata to /tmp/snapper_thinp_dump.6257.16216
thin_dump /dev/mapper/snapper_thinp-POOL_tmeta > /tmp/snapper_thinp_dump.6257.16216

Current tmeta device: /dev/sdb2
Restoring valid mda to new device
thin_restore -i /tmp/snapper_thinp_dump.6257.16216 -o /dev/snapper_thinp/newtmeta

Corrupting pool meta device (/dev/mapper/snapper_thinp-POOL_tmeta)
dd if=/dev/zero of=/dev/mapper/snapper_thinp-POOL_tmeta count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000302017 s, 1.7 MB/s
Verifying that pool meta device is now corrupt
thin_check /dev/mapper/snapper_thinp-POOL_tmeta
examining superblock
  superblock is corrupt
    bad checksum in superblock
  WARNING: Integrity check of metadata for thin pool snapper_thinp/POOL failed.

Swap in new _tmeta device
lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL
New swapped tmeta device: /dev/sdc3

root@harding-02 ~]# lvs -a -o +devices
 LV              VG            Attr       LSize  Pool Origin Data%  Devices
 POOL            snapper_thinp twi---t---  1.00g                    POOL_tdata(0)
 [POOL_tdata]    snapper_thinp Twi-------  1.00g                    /dev/sdc3(1)
 [POOL_tmeta]    snapper_thinp ewi-------  8.00m                    /dev/sdc3(257)
 [lvol0_pmspare] snapper_thinp ewi-------  4.00m                    /dev/sdc3(0)
 origin          snapper_thinp Vwi---t---  1.00g POOL
 other1          snapper_thinp Vwi---t---  1.00g POOL
 other2          snapper_thinp Vwi---t---  1.00g POOL
 other3          snapper_thinp Vwi---t---  1.00g POOL
 other4          snapper_thinp Vwi---t---  1.00g POOL
 other5          snapper_thinp Vwi---t---  1.00g POOL
 snap            snapper_thinp Vwi---t--k  1.00g POOL origin
[root@harding-02 ~]# vgchange -an snapper_thinp
 0 logical volume(s) in volume group "snapper_thinp" now active
[root@harding-02 ~]# lvremove -f /dev/snapper_thinp/snap
 Thin pool transaction_id=0, while expected: 7.
 Unable to deactivate open snapper_thinp-POOL_tmeta (253:2)
 Unable to deactivate open snapper_thinp-POOL_tdata (253:3)
 Failed to deactivate snapper_thinp-POOL-tpool
 Failed to update thin pool POOL.
[root@harding-02 ~]# lvs -a -o +devices
 LV              VG            Attr       LSize  Pool Origin Data%  Devices
 POOL            snapper_thinp twi-i-t---  1.00g               0.00 POOL_tdata(0)
 [POOL_tdata]    snapper_thinp Twi-ao----  1.00g                    /dev/sdc3(1)
 [POOL_tmeta]    snapper_thinp ewi-ao----  8.00m                    /dev/sdc3(257)
 [lvol0_pmspare] snapper_thinp ewi-------  4.00m                    /dev/sdc3(0)
 origin          snapper_thinp Vwi---t---  1.00g POOL
 other1          snapper_thinp Vwi---t---  1.00g POOL
 other2          snapper_thinp Vwi---t---  1.00g POOL
 other3          snapper_thinp Vwi---t---  1.00g POOL
 other4          snapper_thinp Vwi---t---  1.00g POOL
 other5          snapper_thinp Vwi---t---  1.00g POOL
[root@harding-02 ~]# lvremove -f /dev/snapper_thinp/origin
 Thin pool transaction_id=0, while expected: 7.
 Unable to deactivate open snapper_thinp-POOL_tdata (253:3)
 Unable to deactivate open snapper_thinp-POOL_tmeta (253:2)
 Failed to deactivate snapper_thinp-POOL-tpool
 Failed to resume POOL.
 Failed to update thin pool POOL.
[root@harding-02 ~]# lvs -a -o +devices
 LV              VG            Attr       LSize  Pool Origin Data%  Devices
 POOL            snapper_thinp twi-i-t---  1.00g               0.00 POOL_tdata(0)
 [POOL_tdata]    snapper_thinp Twi-ao----  1.00g                    /dev/sdc3(1)
 [POOL_tmeta]    snapper_thinp ewi-ao----  8.00m                    /dev/sdc3(257)
 [lvol0_pmspare] snapper_thinp ewi-------  4.00m                    /dev/sdc3(0)
 origin          snapper_thinp Vwi---t---  1.00g POOL
 other1          snapper_thinp Vwi---t---  1.00g POOL
 other2          snapper_thinp Vwi---t---  1.00g POOL
 other3          snapper_thinp Vwi---t---  1.00g POOL
 other4          snapper_thinp Vwi---t---  1.00g POOL
 other5          snapper_thinp Vwi---t---  1.00g POOL

# NOTE: the origin volume still exists above.


Version-Release number of selected component (if applicable):
2.6.32-410.el6.x86_64
lvm2-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
lvm2-libs-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
lvm2-cluster-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
udev-147-2.50.el6    BUILT: Fri Oct 11 05:58:10 CDT 2013
device-mapper-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
device-mapper-libs-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
device-mapper-event-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
device-mapper-event-libs-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
cmirror-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013


How reproducible:
Everytime
Comment 2 Zdenek Kabelac 2013-10-22 03:45:38 EDT
As mentioned in Bug #1007074  comment 13 - removal of damaged thin pool is still unpleasant user experience - since it requires manual intervention.

Tool currently doesn't copy with stacked removal of whole tree of device at once - this will need to be addressed upstream in some nearby future releases of lvm2.

So for now - metadata manual text editor manipulation is needed (and need to be documented) - Minor help here is - if the thin-pool related LVs are the only LVs in the VG - then removal of all related dm table lines and pvremove -ff works.

This BZ will be used to track steps forward in this damaged thin-pool removal process.
Comment 3 Zdenek Kabelac 2014-11-26 15:41:04 EST
With version 2.02.112 we support removal of 'broken' thin-pool with '-ff'.
Double --force option will ignore errors with thin-pool (& cache-pool) manipulation.

lvremove -ff  vg

Should cleanup any LV in a VG with damaged thin-pool included.
Comment 5 Corey Marthaler 2015-05-08 18:22:40 EDT
Marking verified in the latest rpms. The sequence of commands in comment #0 no longer result in problems.

2.6.32-558.el6.x86_64
lvm2-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
lvm2-libs-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
lvm2-cluster-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
udev-147-2.62.el6    BUILT: Thu Apr 23 05:44:37 CDT 2015
device-mapper-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-libs-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-event-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-event-libs-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-persistent-data-0.3.2-1.el6    BUILT: Fri Apr  4 08:43:06 CDT 2014
cmirror-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015


[root@host-075 ~]# lvcreate -n newtmeta -L 8M snapper_thinp
[root@host-075 ~]# thin_dump /dev/mapper/snapper_thinp-POOL_tmeta > /tmp/snapper_thinp_dump.6257.16216
[root@host-075 ~]# thin_restore -i /tmp/snapper_thinp_dump.6257.16216 -o /dev/snapper_thinp/newtmeta
[root@host-075 ~]# dd if=/dev/zero of=/dev/mapper/snapper_thinp-POOL_tmeta count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00180128 s, 284 kB/s
[root@host-075 ~]# thin_check /dev/mapper/snapper_thinp-POOL_tmeta
examining superblock
  superblock is corrupt
    bad checksum in superblock

[root@host-075 ~]# lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL
  Cannot convert pool snapper_thinp/POOL with active volumes.
[root@host-075 ~]# vgchange -an snapper_thinp
  0 logical volume(s) in volume group "snapper_thinp" now active

[root@host-075 ~]# lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL
  Converted snapper_thinp/POOL to thin pool.

[root@host-075 ~]# vgchange -an snapper_thinp
  0 logical volume(s) in volume group "snapper_thinp" now active

[root@host-075 ~]# lvremove -f /dev/snapper_thinp/snap
  Logical volume "snap" successfully removed
[root@host-075 ~]# lvs -a -o +devices
  LV              Attr       LSize   Pool Origin Devices
  POOL            twi---t---   1.00g             POOL_tdata(0)
  [POOL_tdata]    Twi-------   1.00g             /dev/sda1(1)
  [POOL_tmeta]    ewi-------   8.00m             /dev/sda1(257)
  [lvol0_pmspare] ewi-------   8.00m             /dev/sda1(0)
  [lvol0_pmspare] ewi-------   8.00m             /dev/sda1(259)
  newtmeta        -wi-------   4.00m             /dev/sdh1(0)
  origin          Vwi---t---   1.00g POOL
  other1          Vwi---t---   1.00g POOL
  other2          Vwi---t---   1.00g POOL
  other3          Vwi---t---   1.00g POOL
  other4          Vwi---t---   1.00g POOL
  other5          Vwi---t---   1.00g POOL

[root@host-075 ~]# lvremove -f /dev/snapper_thinp/origin
  Logical volume "origin" successfully removed
[root@host-075 ~]# lvs -a -o +devices
  LV              Attr       LSize   Pool Origin Devices
  POOL            twi---t---   1.00g             POOL_tdata(0)
  [POOL_tdata]    Twi-------   1.00g             /dev/sda1(1)
  [POOL_tmeta]    ewi-------   8.00m             /dev/sda1(257)
  [lvol0_pmspare] ewi-------   8.00m             /dev/sda1(0)
  [lvol0_pmspare] ewi-------   8.00m             /dev/sda1(259)
  newtmeta        -wi-------   4.00m             /dev/sdh1(0)
  other1          Vwi---t---   1.00g POOL
  other2          Vwi---t---   1.00g POOL
  other3          Vwi---t---   1.00g POOL
  other4          Vwi---t---   1.00g POOL
  other5          Vwi---t---   1.00g POOL
Comment 6 errata-xmlrpc 2015-07-22 03:37:18 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1411.html

Note You need to log in before you can comment on or make changes to this bug.