Hide Forgot
Description of problem: I have yet to successfully remove thin snap or thin origin volumes after corrupting and swapping pool meta devices. SCENARIO - [swap_deactive_thin_pool_meta_device_w_linear] Swap a _tmeta device with newly created linear LV while pool is deactivated Making origin volume lvcreate --thinpool POOL --zero n -L 1G snapper_thinp Sanity checking pool device metadata (thin_check /dev/mapper/snapper_thinp-POOL_tmeta) examining superblock examining devices tree examining mapping tree lvcreate --virtualsize 1G -T snapper_thinp/POOL -n origin lvcreate -V 1G -T snapper_thinp/POOL -n other1 lvcreate --virtualsize 1G -T snapper_thinp/POOL -n other2 lvcreate -V 1G -T snapper_thinp/POOL -n other3 lvcreate --virtualsize 1G -T snapper_thinp/POOL -n other4 lvcreate -V 1G -T snapper_thinp/POOL -n other5 Making snapshot of origin volume lvcreate -K -s /dev/snapper_thinp/origin -n snap Create new device to swap in as the new _tmeta device Dumping current pool metadata to /tmp/snapper_thinp_dump.6257.16216 thin_dump /dev/mapper/snapper_thinp-POOL_tmeta > /tmp/snapper_thinp_dump.6257.16216 Current tmeta device: /dev/sdb2 Restoring valid mda to new device thin_restore -i /tmp/snapper_thinp_dump.6257.16216 -o /dev/snapper_thinp/newtmeta Corrupting pool meta device (/dev/mapper/snapper_thinp-POOL_tmeta) dd if=/dev/zero of=/dev/mapper/snapper_thinp-POOL_tmeta count=1 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.000302017 s, 1.7 MB/s Verifying that pool meta device is now corrupt thin_check /dev/mapper/snapper_thinp-POOL_tmeta examining superblock superblock is corrupt bad checksum in superblock WARNING: Integrity check of metadata for thin pool snapper_thinp/POOL failed. Swap in new _tmeta device lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL New swapped tmeta device: /dev/sdc3 root@harding-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Devices POOL snapper_thinp twi---t--- 1.00g POOL_tdata(0) [POOL_tdata] snapper_thinp Twi------- 1.00g /dev/sdc3(1) [POOL_tmeta] snapper_thinp ewi------- 8.00m /dev/sdc3(257) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdc3(0) origin snapper_thinp Vwi---t--- 1.00g POOL other1 snapper_thinp Vwi---t--- 1.00g POOL other2 snapper_thinp Vwi---t--- 1.00g POOL other3 snapper_thinp Vwi---t--- 1.00g POOL other4 snapper_thinp Vwi---t--- 1.00g POOL other5 snapper_thinp Vwi---t--- 1.00g POOL snap snapper_thinp Vwi---t--k 1.00g POOL origin [root@harding-02 ~]# vgchange -an snapper_thinp 0 logical volume(s) in volume group "snapper_thinp" now active [root@harding-02 ~]# lvremove -f /dev/snapper_thinp/snap Thin pool transaction_id=0, while expected: 7. Unable to deactivate open snapper_thinp-POOL_tmeta (253:2) Unable to deactivate open snapper_thinp-POOL_tdata (253:3) Failed to deactivate snapper_thinp-POOL-tpool Failed to update thin pool POOL. [root@harding-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Devices POOL snapper_thinp twi-i-t--- 1.00g 0.00 POOL_tdata(0) [POOL_tdata] snapper_thinp Twi-ao---- 1.00g /dev/sdc3(1) [POOL_tmeta] snapper_thinp ewi-ao---- 8.00m /dev/sdc3(257) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdc3(0) origin snapper_thinp Vwi---t--- 1.00g POOL other1 snapper_thinp Vwi---t--- 1.00g POOL other2 snapper_thinp Vwi---t--- 1.00g POOL other3 snapper_thinp Vwi---t--- 1.00g POOL other4 snapper_thinp Vwi---t--- 1.00g POOL other5 snapper_thinp Vwi---t--- 1.00g POOL [root@harding-02 ~]# lvremove -f /dev/snapper_thinp/origin Thin pool transaction_id=0, while expected: 7. Unable to deactivate open snapper_thinp-POOL_tdata (253:3) Unable to deactivate open snapper_thinp-POOL_tmeta (253:2) Failed to deactivate snapper_thinp-POOL-tpool Failed to resume POOL. Failed to update thin pool POOL. [root@harding-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Devices POOL snapper_thinp twi-i-t--- 1.00g 0.00 POOL_tdata(0) [POOL_tdata] snapper_thinp Twi-ao---- 1.00g /dev/sdc3(1) [POOL_tmeta] snapper_thinp ewi-ao---- 8.00m /dev/sdc3(257) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdc3(0) origin snapper_thinp Vwi---t--- 1.00g POOL other1 snapper_thinp Vwi---t--- 1.00g POOL other2 snapper_thinp Vwi---t--- 1.00g POOL other3 snapper_thinp Vwi---t--- 1.00g POOL other4 snapper_thinp Vwi---t--- 1.00g POOL other5 snapper_thinp Vwi---t--- 1.00g POOL # NOTE: the origin volume still exists above. Version-Release number of selected component (if applicable): 2.6.32-410.el6.x86_64 lvm2-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 lvm2-libs-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 lvm2-cluster-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 udev-147-2.50.el6 BUILT: Fri Oct 11 05:58:10 CDT 2013 device-mapper-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 device-mapper-libs-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 device-mapper-event-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 device-mapper-event-libs-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 cmirror-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 How reproducible: Everytime
As mentioned in Bug #1007074 comment 13 - removal of damaged thin pool is still unpleasant user experience - since it requires manual intervention. Tool currently doesn't copy with stacked removal of whole tree of device at once - this will need to be addressed upstream in some nearby future releases of lvm2. So for now - metadata manual text editor manipulation is needed (and need to be documented) - Minor help here is - if the thin-pool related LVs are the only LVs in the VG - then removal of all related dm table lines and pvremove -ff works. This BZ will be used to track steps forward in this damaged thin-pool removal process.
With version 2.02.112 we support removal of 'broken' thin-pool with '-ff'. Double --force option will ignore errors with thin-pool (& cache-pool) manipulation. lvremove -ff vg Should cleanup any LV in a VG with damaged thin-pool included.
Marking verified in the latest rpms. The sequence of commands in comment #0 no longer result in problems. 2.6.32-558.el6.x86_64 lvm2-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 lvm2-libs-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 lvm2-cluster-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 udev-147-2.62.el6 BUILT: Thu Apr 23 05:44:37 CDT 2015 device-mapper-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-libs-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-event-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-event-libs-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-persistent-data-0.3.2-1.el6 BUILT: Fri Apr 4 08:43:06 CDT 2014 cmirror-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 [root@host-075 ~]# lvcreate -n newtmeta -L 8M snapper_thinp [root@host-075 ~]# thin_dump /dev/mapper/snapper_thinp-POOL_tmeta > /tmp/snapper_thinp_dump.6257.16216 [root@host-075 ~]# thin_restore -i /tmp/snapper_thinp_dump.6257.16216 -o /dev/snapper_thinp/newtmeta [root@host-075 ~]# dd if=/dev/zero of=/dev/mapper/snapper_thinp-POOL_tmeta count=1 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00180128 s, 284 kB/s [root@host-075 ~]# thin_check /dev/mapper/snapper_thinp-POOL_tmeta examining superblock superblock is corrupt bad checksum in superblock [root@host-075 ~]# lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL Cannot convert pool snapper_thinp/POOL with active volumes. [root@host-075 ~]# vgchange -an snapper_thinp 0 logical volume(s) in volume group "snapper_thinp" now active [root@host-075 ~]# lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL Converted snapper_thinp/POOL to thin pool. [root@host-075 ~]# vgchange -an snapper_thinp 0 logical volume(s) in volume group "snapper_thinp" now active [root@host-075 ~]# lvremove -f /dev/snapper_thinp/snap Logical volume "snap" successfully removed [root@host-075 ~]# lvs -a -o +devices LV Attr LSize Pool Origin Devices POOL twi---t--- 1.00g POOL_tdata(0) [POOL_tdata] Twi------- 1.00g /dev/sda1(1) [POOL_tmeta] ewi------- 8.00m /dev/sda1(257) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(0) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(259) newtmeta -wi------- 4.00m /dev/sdh1(0) origin Vwi---t--- 1.00g POOL other1 Vwi---t--- 1.00g POOL other2 Vwi---t--- 1.00g POOL other3 Vwi---t--- 1.00g POOL other4 Vwi---t--- 1.00g POOL other5 Vwi---t--- 1.00g POOL [root@host-075 ~]# lvremove -f /dev/snapper_thinp/origin Logical volume "origin" successfully removed [root@host-075 ~]# lvs -a -o +devices LV Attr LSize Pool Origin Devices POOL twi---t--- 1.00g POOL_tdata(0) [POOL_tdata] Twi------- 1.00g /dev/sda1(1) [POOL_tmeta] ewi------- 8.00m /dev/sda1(257) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(0) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(259) newtmeta -wi------- 4.00m /dev/sdh1(0) other1 Vwi---t--- 1.00g POOL other2 Vwi---t--- 1.00g POOL other3 Vwi---t--- 1.00g POOL other4 Vwi---t--- 1.00g POOL other5 Vwi---t--- 1.00g POOL
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1411.html