| Summary: | remove issues after pool meta corruption and restoration | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Corey Marthaler <cmarthal> |
| Component: | lvm2 | Assignee: | Zdenek Kabelac <zkabelac> |
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.5 | CC: | agk, dwysocha, heinzm, jbrassow, msnitzer, nperic, prajnoha, prockai, thornber, zkabelac |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | lvm2-2.02.117-1.el6 | Doc Type: | Bug Fix |
| Doc Text: |
If the thin-pool was damaged and not repaired, the lvremove failed to remove such pool, as it required to activated such volume. If the user has no intention to preserve such broken pool, the tool now supports double --force --force option to proceed with removal of such pool volume.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-07-22 07:37:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
As mentioned in Bug #1007074 comment 13 - removal of damaged thin pool is still unpleasant user experience - since it requires manual intervention. Tool currently doesn't copy with stacked removal of whole tree of device at once - this will need to be addressed upstream in some nearby future releases of lvm2. So for now - metadata manual text editor manipulation is needed (and need to be documented) - Minor help here is - if the thin-pool related LVs are the only LVs in the VG - then removal of all related dm table lines and pvremove -ff works. This BZ will be used to track steps forward in this damaged thin-pool removal process. With version 2.02.112 we support removal of 'broken' thin-pool with '-ff'. Double --force option will ignore errors with thin-pool (& cache-pool) manipulation. lvremove -ff vg Should cleanup any LV in a VG with damaged thin-pool included. Marking verified in the latest rpms. The sequence of commands in comment #0 no longer result in problems. 2.6.32-558.el6.x86_64 lvm2-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 lvm2-libs-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 lvm2-cluster-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 udev-147-2.62.el6 BUILT: Thu Apr 23 05:44:37 CDT 2015 device-mapper-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-libs-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-event-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-event-libs-1.02.95-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 device-mapper-persistent-data-0.3.2-1.el6 BUILT: Fri Apr 4 08:43:06 CDT 2014 cmirror-2.02.118-2.el6 BUILT: Wed Apr 15 06:34:08 CDT 2015 [root@host-075 ~]# lvcreate -n newtmeta -L 8M snapper_thinp [root@host-075 ~]# thin_dump /dev/mapper/snapper_thinp-POOL_tmeta > /tmp/snapper_thinp_dump.6257.16216 [root@host-075 ~]# thin_restore -i /tmp/snapper_thinp_dump.6257.16216 -o /dev/snapper_thinp/newtmeta [root@host-075 ~]# dd if=/dev/zero of=/dev/mapper/snapper_thinp-POOL_tmeta count=1 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00180128 s, 284 kB/s [root@host-075 ~]# thin_check /dev/mapper/snapper_thinp-POOL_tmeta examining superblock superblock is corrupt bad checksum in superblock [root@host-075 ~]# lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL Cannot convert pool snapper_thinp/POOL with active volumes. [root@host-075 ~]# vgchange -an snapper_thinp 0 logical volume(s) in volume group "snapper_thinp" now active [root@host-075 ~]# lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL Converted snapper_thinp/POOL to thin pool. [root@host-075 ~]# vgchange -an snapper_thinp 0 logical volume(s) in volume group "snapper_thinp" now active [root@host-075 ~]# lvremove -f /dev/snapper_thinp/snap Logical volume "snap" successfully removed [root@host-075 ~]# lvs -a -o +devices LV Attr LSize Pool Origin Devices POOL twi---t--- 1.00g POOL_tdata(0) [POOL_tdata] Twi------- 1.00g /dev/sda1(1) [POOL_tmeta] ewi------- 8.00m /dev/sda1(257) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(0) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(259) newtmeta -wi------- 4.00m /dev/sdh1(0) origin Vwi---t--- 1.00g POOL other1 Vwi---t--- 1.00g POOL other2 Vwi---t--- 1.00g POOL other3 Vwi---t--- 1.00g POOL other4 Vwi---t--- 1.00g POOL other5 Vwi---t--- 1.00g POOL [root@host-075 ~]# lvremove -f /dev/snapper_thinp/origin Logical volume "origin" successfully removed [root@host-075 ~]# lvs -a -o +devices LV Attr LSize Pool Origin Devices POOL twi---t--- 1.00g POOL_tdata(0) [POOL_tdata] Twi------- 1.00g /dev/sda1(1) [POOL_tmeta] ewi------- 8.00m /dev/sda1(257) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(0) [lvol0_pmspare] ewi------- 8.00m /dev/sda1(259) newtmeta -wi------- 4.00m /dev/sdh1(0) other1 Vwi---t--- 1.00g POOL other2 Vwi---t--- 1.00g POOL other3 Vwi---t--- 1.00g POOL other4 Vwi---t--- 1.00g POOL other5 Vwi---t--- 1.00g POOL Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1411.html |
Description of problem: I have yet to successfully remove thin snap or thin origin volumes after corrupting and swapping pool meta devices. SCENARIO - [swap_deactive_thin_pool_meta_device_w_linear] Swap a _tmeta device with newly created linear LV while pool is deactivated Making origin volume lvcreate --thinpool POOL --zero n -L 1G snapper_thinp Sanity checking pool device metadata (thin_check /dev/mapper/snapper_thinp-POOL_tmeta) examining superblock examining devices tree examining mapping tree lvcreate --virtualsize 1G -T snapper_thinp/POOL -n origin lvcreate -V 1G -T snapper_thinp/POOL -n other1 lvcreate --virtualsize 1G -T snapper_thinp/POOL -n other2 lvcreate -V 1G -T snapper_thinp/POOL -n other3 lvcreate --virtualsize 1G -T snapper_thinp/POOL -n other4 lvcreate -V 1G -T snapper_thinp/POOL -n other5 Making snapshot of origin volume lvcreate -K -s /dev/snapper_thinp/origin -n snap Create new device to swap in as the new _tmeta device Dumping current pool metadata to /tmp/snapper_thinp_dump.6257.16216 thin_dump /dev/mapper/snapper_thinp-POOL_tmeta > /tmp/snapper_thinp_dump.6257.16216 Current tmeta device: /dev/sdb2 Restoring valid mda to new device thin_restore -i /tmp/snapper_thinp_dump.6257.16216 -o /dev/snapper_thinp/newtmeta Corrupting pool meta device (/dev/mapper/snapper_thinp-POOL_tmeta) dd if=/dev/zero of=/dev/mapper/snapper_thinp-POOL_tmeta count=1 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.000302017 s, 1.7 MB/s Verifying that pool meta device is now corrupt thin_check /dev/mapper/snapper_thinp-POOL_tmeta examining superblock superblock is corrupt bad checksum in superblock WARNING: Integrity check of metadata for thin pool snapper_thinp/POOL failed. Swap in new _tmeta device lvconvert --yes --poolmetadata snapper_thinp/newtmeta --thinpool snapper_thinp/POOL New swapped tmeta device: /dev/sdc3 root@harding-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Devices POOL snapper_thinp twi---t--- 1.00g POOL_tdata(0) [POOL_tdata] snapper_thinp Twi------- 1.00g /dev/sdc3(1) [POOL_tmeta] snapper_thinp ewi------- 8.00m /dev/sdc3(257) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdc3(0) origin snapper_thinp Vwi---t--- 1.00g POOL other1 snapper_thinp Vwi---t--- 1.00g POOL other2 snapper_thinp Vwi---t--- 1.00g POOL other3 snapper_thinp Vwi---t--- 1.00g POOL other4 snapper_thinp Vwi---t--- 1.00g POOL other5 snapper_thinp Vwi---t--- 1.00g POOL snap snapper_thinp Vwi---t--k 1.00g POOL origin [root@harding-02 ~]# vgchange -an snapper_thinp 0 logical volume(s) in volume group "snapper_thinp" now active [root@harding-02 ~]# lvremove -f /dev/snapper_thinp/snap Thin pool transaction_id=0, while expected: 7. Unable to deactivate open snapper_thinp-POOL_tmeta (253:2) Unable to deactivate open snapper_thinp-POOL_tdata (253:3) Failed to deactivate snapper_thinp-POOL-tpool Failed to update thin pool POOL. [root@harding-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Devices POOL snapper_thinp twi-i-t--- 1.00g 0.00 POOL_tdata(0) [POOL_tdata] snapper_thinp Twi-ao---- 1.00g /dev/sdc3(1) [POOL_tmeta] snapper_thinp ewi-ao---- 8.00m /dev/sdc3(257) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdc3(0) origin snapper_thinp Vwi---t--- 1.00g POOL other1 snapper_thinp Vwi---t--- 1.00g POOL other2 snapper_thinp Vwi---t--- 1.00g POOL other3 snapper_thinp Vwi---t--- 1.00g POOL other4 snapper_thinp Vwi---t--- 1.00g POOL other5 snapper_thinp Vwi---t--- 1.00g POOL [root@harding-02 ~]# lvremove -f /dev/snapper_thinp/origin Thin pool transaction_id=0, while expected: 7. Unable to deactivate open snapper_thinp-POOL_tdata (253:3) Unable to deactivate open snapper_thinp-POOL_tmeta (253:2) Failed to deactivate snapper_thinp-POOL-tpool Failed to resume POOL. Failed to update thin pool POOL. [root@harding-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Devices POOL snapper_thinp twi-i-t--- 1.00g 0.00 POOL_tdata(0) [POOL_tdata] snapper_thinp Twi-ao---- 1.00g /dev/sdc3(1) [POOL_tmeta] snapper_thinp ewi-ao---- 8.00m /dev/sdc3(257) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdc3(0) origin snapper_thinp Vwi---t--- 1.00g POOL other1 snapper_thinp Vwi---t--- 1.00g POOL other2 snapper_thinp Vwi---t--- 1.00g POOL other3 snapper_thinp Vwi---t--- 1.00g POOL other4 snapper_thinp Vwi---t--- 1.00g POOL other5 snapper_thinp Vwi---t--- 1.00g POOL # NOTE: the origin volume still exists above. Version-Release number of selected component (if applicable): 2.6.32-410.el6.x86_64 lvm2-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 lvm2-libs-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 lvm2-cluster-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 udev-147-2.50.el6 BUILT: Fri Oct 11 05:58:10 CDT 2013 device-mapper-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 device-mapper-libs-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 device-mapper-event-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 device-mapper-event-libs-1.02.79-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 cmirror-2.02.100-6.el6 BUILT: Wed Oct 16 07:26:00 CDT 2013 How reproducible: Everytime