Bug 1131777

Summary: LVM cache: If writethrough, allow abandoning cachepool (lvconvert --uncache) if it has failed, and is also inactive
Product: Red Hat Enterprise Linux 7 Reporter: Jonathan Earl Brassow <jbrassow>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: Cache Logical Volumes QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact: Steven J. Levine <slevine>
Severity: unspecified    
Priority: unspecified CC: agk, cmarthal, coughlan, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, rbednar, slevine, zkabelac
Version: 7.1   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: lvm2-2.02.161-2.el7 Doc Type: Enhancement
Doc Text:
Support for ability to uncouple a cache pool from a logical volume LVM now has the ability to uncouple a cache pool from a logical volume if a device in the cache pool has failed. Previously, this type of failure would require manual intervention and complicated alterations to LVM metadata in order to separate the cache pool from the origin logical volume. To uncouple a logical volume from its cache-pool use the following command: # lvconvert --uncache *vg*/*lv* Note the following limitations: * The cache logical volume must be inactive (may require a reboot) * A `writeback` cache requires the "--force" option due to the possibility of abandoning data lost to failure.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 04:08:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1295577, 1313485    

Description Jonathan Earl Brassow 2014-08-20 04:24:53 UTC 
If 'writethrough' caching is being employed for a cache LV and the cache-pool dies (either cacheDataLV or cacheMetaLV) in an unrecoverable way, allow the repair tools to abandon the cache-pool and convert the origin back to a non-cache LV.
Comment 2 Marian Csontos 2014-08-20 09:15:24 UTC 
In my opinion with enough "--force" one should be able to convert writeback devices too. We do not want users to edit metadata by hand, do we? Recovery tools should be still able to save at least some data...
Comment 7 Zdenek Kabelac 2016-07-22 12:29:17 UTC 
Original commit:

https://www.redhat.com/archives/lvm-devel/2016-March/msg00090.html


Fixing commit for recent upstream changes:

https://www.redhat.com/archives/lvm-devel/2016-July/msg00193.html
Comment 10 Zdenek Kabelac 2016-09-19 13:11:32 UTC 
There is not yet ANY support for repair of broken  cache volumes.

'lvconvert --repair' on a cache volumes is not yet driven by lvm2 and needs manual steps with replacement of cache-pool metadata LV with a fixed version.

In some future (hopefully near-by) we will provide support for offline decommission of cached blocks as well as some automated repair of (slightly0 damaged cache metadata volume.
Comment 11 Corey Marthaler 2016-09-19 21:15:37 UTC 
With out knowing the specific writethrough case that this was intended to fix, or the devel testing results proving that it worked, I attempted the following seven cases and only two of them passed. So marking this FailedQA for now.



# scenario 1: fail the cache pool device, attempt an uncache: FAILS
[root@host-117 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sda1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sda1(1024)       

[root@host-117 ~]# echo offline > /sys/block/sda/device/state
[root@host-117 ~]# lvconvert --uncache cache_sanity/display_cache
  /dev/sda1: read failed after 0 of 512 at 4096: Input/output error
  WARNING: Failed to write an MDA of VG cache_sanity.
[DEADLOCK]


# scenario 2: fail the cache pool device, attempt a pvscan: FAILS
[root@host-116 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sdd1(1024)       
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdf1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdf1(1024)       

[root@host-116 ~]# echo offline > /sys/block/sdf/device/state
[root@host-116 ~]# pvscan --cache
[DEADLOCK]


# scenario 3: fail the cache pool device, attempt a splitcache, then attempt an uncache: PASSES
[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   8.66   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sdf1(1024)       
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   8.66   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdd1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdd1(1024)       
[root@host-118 ~]# lvconvert --yes --splitcache cache_sanity/display_cache
  /dev/sdd1: read failed after 0 of 512 at 4096: Input/output error
  Failed to write VG cache_sanity.

[441927.867015] sd 4:0:0:1: rejecting I/O to offline device
[441927.867015] sd 4:0:0:1: rejecting I/O to offline device
[441927.874658] metadata_operation_failed: 384 callbacks suppressed
[441927.876002] device-mapper: cache: 253:2: metadata operation 'dm_cache_commit' failed: error = -5
[441927.877705] device-mapper: cache: 253:2: aborting current metadata transaction
[441927.879073] sd 4:0:0:1: rejecting I/O to offline device
[441927.881134] sd 4:0:0:1: rejecting I/O to offline device
[441927.881774] device-mapper: cache: 253:2: failed to abort metadata transaction
[441927.882603] device-mapper: cache: unable to read needs_check flag, setting failure mode
[441927.883701] device-mapper: cache: 253:2: switching cache to fail mode
[441927.884599] device-mapper: cache: unable to read needs_check flag, setting failure mode
[441927.885918] device-mapper: cache: 253:2: could not write cache metadata
[441927.899748] sd 4:0:0:1: rejecting I/O to offline device

[root@host-118 ~]# lvconvert --yes --uncache cache_sanity/display_cache
  WARNING: Skippping flush for failed cache cache_sanity/display_cache.
  /dev/sdd1: read failed after 0 of 512 at 4096: Input/output error
  WARNING: Failed to write an MDA of VG cache_sanity.
  WARNING: Failed to write an MDA of VG cache_sanity.
  Logical volume "pool" successfully removed
  Logical volume cache_sanity/display_cache is not cached.
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV vFLT6Z-sj38-ivEh-lC3n-VkWV-5V6t-OkUYZD not found or rejected by a filter.
  LV            VG            Attr       LSize   Pool Origin Data%  Meta%  Cpy%Sync Devices       
  display_cache cache_sanity  -wi-a-----   4.00g                                    /dev/sdf1(0)  


# scenario 4: fail the cache pool device, attempt a write to the cache origin: FAILS
[root@host-117 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sda1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sda1(1024)       
[root@host-117 ~]#  echo offline > /sys/block/sda/device/state
[root@host-117 ~]# dd if=/dev/zero of=/dev/cache_sanity/display_cache count=1 bs=1M
[DEADLOCK]



# scenario 5: corrupt the cache pool device by writing zeros over it, attempt an uncache: FAILS
[root@host-116 ~]# lvs -a -o +devices
  WARNING: Device for PV ZFKtOZ-eSBT-eAdB-PBsD-hWOs-sqRL-0TcP3K not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi-a-C-p-   4.00g [corrupt_pool_meta]        0.00   2.31   0.00     corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi-aoC---   4.00g                                                   /dev/sda1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                            0.00   2.31   0.00     corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-ao--p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-ao--p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sdd1(0)              

[root@host-116 ~]# lvconvert --uncache cache_sanity/corigin
  WARNING: Device for PV ZFKtOZ-eSBT-eAdB-PBsD-hWOs-sqRL-0TcP3K not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  WARNING: Cache pool data logical volume cache_sanity/corrupt_pool_meta_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/corrupt_pool_meta_cmeta is missing.
Do you really want to uncache cache_sanity/corigin with missing LVs? [y/n]: y
[DEADLOCK]

Sep 19 15:11:59 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: migration_failure: 7719 callbacks suppressed
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:05 host-116 kernel: INFO: task systemd-udevd:3928 blocked for more than 120 seconds.
Sep 19 15:12:05 host-116 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 19 15:12:05 host-116 kernel: systemd-udevd   D ffff88003b993a18     0  3928    476 0x00000086
Sep 19 15:12:05 host-116 kernel: ffff88003b9938b0 0000000000000046 ffff88003b54edd0 ffff88003b993fd8
Sep 19 15:12:05 host-116 kernel: ffff88003b993fd8 ffff88003b993fd8 ffff88003b54edd0 ffff88003fc16c40
Sep 19 15:12:05 host-116 kernel: 0000000000000000 7fffffffffffffff ffffffff81686b20 ffff88003b993a18
Sep 19 15:12:05 host-116 kernel: Call Trace:
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b20>] ? bit_wait+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff81688aa9>] schedule+0x29/0x70
Sep 19 15:12:05 host-116 kernel: [<ffffffff816864f9>] schedule_timeout+0x239/0x2d0
Sep 19 15:12:05 host-116 kernel: [<ffffffffa03722f0>] ? cache_resume+0x30/0x30 [dm_cache]
Sep 19 15:12:05 host-116 kernel: [<ffffffff81060aef>] ? kvm_clock_get_cycles+0x1f/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b20>] ? bit_wait+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff8168804e>] io_schedule_timeout+0xae/0x130
Sep 19 15:12:05 host-116 kernel: [<ffffffff816880e8>] io_schedule+0x18/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b31>] bit_wait_io+0x11/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff8168684f>] __wait_on_bit_lock+0x5f/0xc0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8117f944>] __lock_page+0x74/0x90
Sep 19 15:12:05 host-116 kernel: [<ffffffff810b1590>] ? wake_bit_function+0x40/0x40
Sep 19 15:12:05 host-116 kernel: [<ffffffff811908ee>] truncate_inode_pages_range+0x6fe/0x740
Sep 19 15:12:05 host-116 kernel: [<ffffffff8118dca1>] ? pagevec_lookup_tag+0x21/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff813170b5>] ? cpumask_next_and+0x35/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff810f943a>] ? smp_call_function_many+0x5a/0x260
Sep 19 15:12:05 host-116 kernel: [<ffffffff81230f40>] ? mark_buffer_async_write+0x20/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff81317199>] ? free_cpumask_var+0x9/0x10
Sep 19 15:12:05 host-116 kernel: [<ffffffff810f982d>] ? on_each_cpu_cond+0xcd/0x180
Sep 19 15:12:05 host-116 kernel: [<ffffffff81231d70>] ? __brelse+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff81190945>] truncate_inode_pages+0x15/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123891f>] kill_bdev+0x2f/0x40
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123a1e4>] __blkdev_put+0x64/0x1a0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123ac7e>] blkdev_put+0x4e/0x140
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123ae25>] blkdev_close+0x25/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff811ff7e9>] __fput+0xe9/0x260
Sep 19 15:12:05 host-116 kernel: [<ffffffff811ffa9e>] ____fput+0xe/0x10
Sep 19 15:12:05 host-116 kernel: [<ffffffff810acbb4>] task_work_run+0xc4/0xe0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8108bb38>] do_exit+0x2d8/0xa40
Sep 19 15:12:05 host-116 kernel: [<ffffffff812193cf>] ? touch_atime+0x12f/0x160
Sep 19 15:12:05 host-116 kernel: [<ffffffff81181fb8>] ? generic_file_aio_read+0x5d8/0x790
Sep 19 15:12:05 host-116 kernel: [<ffffffff8108c31f>] do_group_exit+0x3f/0xa0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8109d390>] get_signal_to_deliver+0x1d0/0x6d0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8102a427>] do_signal+0x57/0x6c0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8102aaef>] do_notify_resume+0x5f/0xb0



# scenario 6: corrupt the cache pool device by writing zeros over it, reactivate in partial mode, attempt an uncache: PASSES
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi-a-C-p-   4.00g [corrupt_pool_meta]        0.00   4.39   0.00     corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi-aoC---   4.00g                                                   /dev/sdf1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                            0.00   4.39   0.00     corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-ao--p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-ao--p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sde1(0)              

[root@host-118 ~]# vgchange -an --partial cache_sanity
  PARTIAL MODE. Incomplete logical volumes will be processed.
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi---C-p-   4.00g [corrupt_pool_meta]                               corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi---C---   4.00g                                                   /dev/sdf1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                                                   corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-----p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-----p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sde1(0)              
[root@host-118 ~]# lvconvert --uncache cache_sanity/corigin
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/corrupt_pool_meta_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/corrupt_pool_meta_cmeta is missing.
Do you really want to uncache cache_sanity/corigin with missing LVs? [y/n]: y
  Logical volume "corrupt_pool_meta" successfully removed
  Logical volume cache_sanity/corigin is not cached.
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  LV      VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  corigin cache_sanity  -wi-------   4.00g                                                     /dev/sdf1(0)  


# scenario 7: fail the cache pool device, attempt to reactivate in partial mode, attempt an uncache: FAILS
[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdc1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdc1(1024)       

[root@host-118 ~]# echo offline > /sys/block/sdc/device/state
[root@host-118 ~]#  vgchange -an --partial cache_sanity
  PARTIAL MODE. Incomplete logical volumes will be processed.
  /dev/mapper/cache_sanity-pool_cmeta: read failed: Input/output error
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  /dev/sdc1: open failed: No such device or address
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi---C---   4.00g [pool]                               display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g                                      pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-------   4.00g                                      /dev/sdc1(0)          
  [pool_cmeta]          cache_sanity  ewi-------  12.00m                                      /dev/sdc1(1024)       

[root@host-118 ~]# lvconvert --uncache cache_sanity/display_cache
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  device-mapper: reload ioctl on (253:2) failed: No such device or address
  Failed to active cache locally cache_sanity/display_cache.
  Failed to uncache cache_sanity/display_cache.
Comment 12 Corey Marthaler 2016-09-20 18:35:20 UTC 
After talking to devel, this fix was intended for just a specific case where the cache origin and pool also happen to be inactive during or immediately just after the pool failure. So, marking this verified in the latest rpms for this specific scenario *only*. I'll file a bug(s) for all the other cache pool failure scenarios that still fail when abandoning is attempted.


3.10.0-501.el7.x86_64
lvm2-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
lvm2-libs-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
lvm2-cluster-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-libs-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-event-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-event-libs-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016



[root@host-116 ~]# systemctl is-active lvm2-lvmetad.service
active

[root@host-116 ~]#  lvs -a -o +devices
  LV                    VG           Attr        LSize  Pool   Origin Data%  Meta%  Cpy%Sync Devices
  display_cache         cache_sanity Cwi-a-C---   4.00g [pool]        0.00   8.66   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity owi-aoC---   4.00g                                      /dev/sde1(0)
  [lvol0_pmspare]       cache_sanity ewi-------  12.00m                                      /dev/sde1(1024)
  [pool]                cache_sanity Cwi---C---   4.00g               0.00   8.66   0.00     pool_cdata(0)
  [pool_cdata]          cache_sanity Cwi-ao----   4.00g                                      /dev/sdf1(0)
  [pool_cmeta]          cache_sanity ewi-ao----  12.00m                                      /dev/sdf1(1024)

[root@host-116 ~]# echo offline > /sys/block/sdf/device/state
[root@host-116 ~]# vgchange -an cache_sanity
  /dev/mapper/cache_sanity-pool_cmeta: read failed: Input/output error
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-116 ~]# pvscan --cache
[root@host-116 ~]# lvs -a -o +devices
  WARNING: Device for PV s38nsq-eiWj-sEcc-r3bX-6TXq-AxyY-OliF1X not found or rejected by a filter.
  LV                    VG           Attr        LSize  Pool   Origin Data%  Meta%  Cpy%Sync Devices
  display_cache         cache_sanity Cwi---C-p-   4.00g [pool]                               display_cache_corig(0)
  [display_cache_corig] cache_sanity owi---C---   4.00g                                      /dev/sde1(0)
  [lvol0_pmspare]       cache_sanity ewi-------  12.00m                                      /dev/sde1(1024)
  [pool]                cache_sanity Cwi---C-p-   4.00g                                      pool_cdata(0)
  [pool_cdata]          cache_sanity Cwi-----p-   4.00g                                      [unknown](0)
  [pool_cmeta]          cache_sanity ewi-----p-  12.00m                                      [unknown](1024)

[root@host-116 ~]# lvconvert --yes --uncache cache_sanity/display_cache
  WARNING: Device for PV s38nsq-eiWj-sEcc-r3bX-6TXq-AxyY-OliF1X not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/pool_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/pool_cmeta is missing.
  Logical volume "pool" successfully removed
  Logical volume cache_sanity/display_cache is not cached.
[root@host-116 ~]# echo $?
0

[root@host-116 ~]# lvs -a -o +devices
  WARNING: Device for PV s38nsq-eiWj-sEcc-r3bX-6TXq-AxyY-OliF1X not found or rejected by a filter.
  LV            VG           Attr        LSize  Pool Origin Data%  Meta%  Cpy%Sync Devices
  display_cache cache_sanity -wi-------   4.00g                                    /dev/sde1(0)
Comment 13 Corey Marthaler 2016-09-20 19:03:33 UTC 
Filed bug 1377850 for the remaining scenarios described in comment #11.
Comment 15 Steven J. Levine 2016-10-05 19:21:37 UTC 
Small formatting updates made to doc text.
Comment 17 errata-xmlrpc 2016-11-04 04:08:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html