Bug 1377850 - LVM cache: allow abandoning cachepool (lvconvert --uncache) if it has failed, remaining scenarios
Summary: LVM cache: allow abandoning cachepool (lvconvert --uncache) if it has failed,...
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Zdenek Kabelac
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Keywords:
: 1379413 (view as bug list)
Depends On: 1379472
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-20 19:01 UTC by Corey Marthaler
Modified: 2019-02-18 13:44 UTC (History)
9 users (show)

(edit)
Clone Of:
(edit)
Last Closed:


Attachments (Terms of Use)

Description Corey Marthaler 2016-09-20 19:01:57 UTC
Description of problem:
This bug is for the remaining cache pool failure issues found in https://bugzilla.redhat.com/show_bug.cgi?id=1131777#c11


# scenario 1: fail the cache pool device, attempt an uncache: FAILS
[root@host-117 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sda1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sda1(1024)       

[root@host-117 ~]# echo offline > /sys/block/sda/device/state
[root@host-117 ~]# lvconvert --uncache cache_sanity/display_cache
  /dev/sda1: read failed after 0 of 512 at 4096: Input/output error
  WARNING: Failed to write an MDA of VG cache_sanity.
[DEADLOCK]


# scenario 2: fail the cache pool device, attempt a pvscan: FAILS
[root@host-116 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sdd1(1024)       
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdf1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdf1(1024)       

[root@host-116 ~]# echo offline > /sys/block/sdf/device/state
[root@host-116 ~]# pvscan --cache
[DEADLOCK]


# scenario 3: fail the cache pool device, attempt a splitcache, then attempt an uncache: PASSES
[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   8.66   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sdf1(1024)       
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   8.66   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdd1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdd1(1024)       
[root@host-118 ~]# lvconvert --yes --splitcache cache_sanity/display_cache
  /dev/sdd1: read failed after 0 of 512 at 4096: Input/output error
  Failed to write VG cache_sanity.

[441927.867015] sd 4:0:0:1: rejecting I/O to offline device
[441927.867015] sd 4:0:0:1: rejecting I/O to offline device
[441927.874658] metadata_operation_failed: 384 callbacks suppressed
[441927.876002] device-mapper: cache: 253:2: metadata operation 'dm_cache_commit' failed: error = -5
[441927.877705] device-mapper: cache: 253:2: aborting current metadata transaction
[441927.879073] sd 4:0:0:1: rejecting I/O to offline device
[441927.881134] sd 4:0:0:1: rejecting I/O to offline device
[441927.881774] device-mapper: cache: 253:2: failed to abort metadata transaction
[441927.882603] device-mapper: cache: unable to read needs_check flag, setting failure mode
[441927.883701] device-mapper: cache: 253:2: switching cache to fail mode
[441927.884599] device-mapper: cache: unable to read needs_check flag, setting failure mode
[441927.885918] device-mapper: cache: 253:2: could not write cache metadata
[441927.899748] sd 4:0:0:1: rejecting I/O to offline device

[root@host-118 ~]# lvconvert --yes --uncache cache_sanity/display_cache
  WARNING: Skippping flush for failed cache cache_sanity/display_cache.
  /dev/sdd1: read failed after 0 of 512 at 4096: Input/output error
  WARNING: Failed to write an MDA of VG cache_sanity.
  WARNING: Failed to write an MDA of VG cache_sanity.
  Logical volume "pool" successfully removed
  Logical volume cache_sanity/display_cache is not cached.
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV vFLT6Z-sj38-ivEh-lC3n-VkWV-5V6t-OkUYZD not found or rejected by a filter.
  LV            VG            Attr       LSize   Pool Origin Data%  Meta%  Cpy%Sync Devices       
  display_cache cache_sanity  -wi-a-----   4.00g                                    /dev/sdf1(0)  


# scenario 4: fail the cache pool device, attempt a write to the cache origin: FAILS
[root@host-117 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sda1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sda1(1024)       
[root@host-117 ~]#  echo offline > /sys/block/sda/device/state
[root@host-117 ~]# dd if=/dev/zero of=/dev/cache_sanity/display_cache count=1 bs=1M
[DEADLOCK]



# scenario 5: corrupt the cache pool device by writing zeros over it, attempt an uncache: FAILS
[root@host-116 ~]# lvs -a -o +devices
  WARNING: Device for PV ZFKtOZ-eSBT-eAdB-PBsD-hWOs-sqRL-0TcP3K not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi-a-C-p-   4.00g [corrupt_pool_meta]        0.00   2.31   0.00     corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi-aoC---   4.00g                                                   /dev/sda1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                            0.00   2.31   0.00     corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-ao--p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-ao--p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sdd1(0)              

[root@host-116 ~]# lvconvert --uncache cache_sanity/corigin
  WARNING: Device for PV ZFKtOZ-eSBT-eAdB-PBsD-hWOs-sqRL-0TcP3K not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  WARNING: Cache pool data logical volume cache_sanity/corrupt_pool_meta_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/corrupt_pool_meta_cmeta is missing.
Do you really want to uncache cache_sanity/corigin with missing LVs? [y/n]: y
[DEADLOCK]

Sep 19 15:11:59 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: migration_failure: 7719 callbacks suppressed
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:05 host-116 kernel: INFO: task systemd-udevd:3928 blocked for more than 120 seconds.
Sep 19 15:12:05 host-116 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 19 15:12:05 host-116 kernel: systemd-udevd   D ffff88003b993a18     0  3928    476 0x00000086
Sep 19 15:12:05 host-116 kernel: ffff88003b9938b0 0000000000000046 ffff88003b54edd0 ffff88003b993fd8
Sep 19 15:12:05 host-116 kernel: ffff88003b993fd8 ffff88003b993fd8 ffff88003b54edd0 ffff88003fc16c40
Sep 19 15:12:05 host-116 kernel: 0000000000000000 7fffffffffffffff ffffffff81686b20 ffff88003b993a18
Sep 19 15:12:05 host-116 kernel: Call Trace:
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b20>] ? bit_wait+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff81688aa9>] schedule+0x29/0x70
Sep 19 15:12:05 host-116 kernel: [<ffffffff816864f9>] schedule_timeout+0x239/0x2d0
Sep 19 15:12:05 host-116 kernel: [<ffffffffa03722f0>] ? cache_resume+0x30/0x30 [dm_cache]
Sep 19 15:12:05 host-116 kernel: [<ffffffff81060aef>] ? kvm_clock_get_cycles+0x1f/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b20>] ? bit_wait+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff8168804e>] io_schedule_timeout+0xae/0x130
Sep 19 15:12:05 host-116 kernel: [<ffffffff816880e8>] io_schedule+0x18/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b31>] bit_wait_io+0x11/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff8168684f>] __wait_on_bit_lock+0x5f/0xc0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8117f944>] __lock_page+0x74/0x90
Sep 19 15:12:05 host-116 kernel: [<ffffffff810b1590>] ? wake_bit_function+0x40/0x40
Sep 19 15:12:05 host-116 kernel: [<ffffffff811908ee>] truncate_inode_pages_range+0x6fe/0x740
Sep 19 15:12:05 host-116 kernel: [<ffffffff8118dca1>] ? pagevec_lookup_tag+0x21/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff813170b5>] ? cpumask_next_and+0x35/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff810f943a>] ? smp_call_function_many+0x5a/0x260
Sep 19 15:12:05 host-116 kernel: [<ffffffff81230f40>] ? mark_buffer_async_write+0x20/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff81317199>] ? free_cpumask_var+0x9/0x10
Sep 19 15:12:05 host-116 kernel: [<ffffffff810f982d>] ? on_each_cpu_cond+0xcd/0x180
Sep 19 15:12:05 host-116 kernel: [<ffffffff81231d70>] ? __brelse+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff81190945>] truncate_inode_pages+0x15/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123891f>] kill_bdev+0x2f/0x40
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123a1e4>] __blkdev_put+0x64/0x1a0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123ac7e>] blkdev_put+0x4e/0x140
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123ae25>] blkdev_close+0x25/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff811ff7e9>] __fput+0xe9/0x260
Sep 19 15:12:05 host-116 kernel: [<ffffffff811ffa9e>] ____fput+0xe/0x10
Sep 19 15:12:05 host-116 kernel: [<ffffffff810acbb4>] task_work_run+0xc4/0xe0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8108bb38>] do_exit+0x2d8/0xa40
Sep 19 15:12:05 host-116 kernel: [<ffffffff812193cf>] ? touch_atime+0x12f/0x160
Sep 19 15:12:05 host-116 kernel: [<ffffffff81181fb8>] ? generic_file_aio_read+0x5d8/0x790
Sep 19 15:12:05 host-116 kernel: [<ffffffff8108c31f>] do_group_exit+0x3f/0xa0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8109d390>] get_signal_to_deliver+0x1d0/0x6d0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8102a427>] do_signal+0x57/0x6c0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8102aaef>] do_notify_resume+0x5f/0xb0



# scenario 6: corrupt the cache pool device by writing zeros over it, reactivate in partial mode, attempt an uncache: PASSES
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi-a-C-p-   4.00g [corrupt_pool_meta]        0.00   4.39   0.00     corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi-aoC---   4.00g                                                   /dev/sdf1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                            0.00   4.39   0.00     corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-ao--p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-ao--p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sde1(0)              

[root@host-118 ~]# vgchange -an --partial cache_sanity
  PARTIAL MODE. Incomplete logical volumes will be processed.
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi---C-p-   4.00g [corrupt_pool_meta]                               corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi---C---   4.00g                                                   /dev/sdf1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                                                   corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-----p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-----p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sde1(0)              
[root@host-118 ~]# lvconvert --uncache cache_sanity/corigin
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/corrupt_pool_meta_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/corrupt_pool_meta_cmeta is missing.
Do you really want to uncache cache_sanity/corigin with missing LVs? [y/n]: y
  Logical volume "corrupt_pool_meta" successfully removed
  Logical volume cache_sanity/corigin is not cached.
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  LV      VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  corigin cache_sanity  -wi-------   4.00g                                                     /dev/sdf1(0)  


# scenario 7: fail the cache pool device, attempt to reactivate in partial mode, attempt an uncache: FAILS
[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdc1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdc1(1024)       

[root@host-118 ~]# echo offline > /sys/block/sdc/device/state
[root@host-118 ~]#  vgchange -an --partial cache_sanity
  PARTIAL MODE. Incomplete logical volumes will be processed.
  /dev/mapper/cache_sanity-pool_cmeta: read failed: Input/output error
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  /dev/sdc1: open failed: No such device or address
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi---C---   4.00g [pool]                               display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g                                      pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-------   4.00g                                      /dev/sdc1(0)          
  [pool_cmeta]          cache_sanity  ewi-------  12.00m                                      /dev/sdc1(1024)       

[root@host-118 ~]# lvconvert --uncache cache_sanity/display_cache
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  device-mapper: reload ioctl on (253:2) failed: No such device or address
  Failed to active cache locally cache_sanity/display_cache.
  Failed to uncache cache_sanity/display_cache.


# Additional scenario 8: the pmspare device is the same PV as the failed cache pool.
[root@host-118 ~]# vgchange -an cache_sanity
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  display_cache         cache_sanity  Cwi---C---   4.00g [pool]                                                display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                                       /dev/sdd1(0)   
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                                       /dev/sda1(1027)
  [pool]                cache_sanity  Cwi---C---   4.00g                                                       pool_cdata(0)
  [pool_cdata]          cache_sanity  Cwi-------   4.00g                                                       /dev/sda1(0)
  [pool_cmeta]          cache_sanity  ewi-------  12.00m                                                       /dev/sda1(1024)

[root@host-118 ~]# echo offline > /sys/block/sda/device/state
[root@host-118 ~]# pvscan --cache
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV nwSfGY-ag3Y-zQeC-C27M-Rjle-gnRt-b9KLXb not found or rejected by a filter.
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  display_cache         cache_sanity  Cwi---C-p-   4.00g [pool]                                                display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                                       /dev/sdd1(0)
  [lvol0_pmspare]       cache_sanity  ewi-----p-  12.00m                                                       [unknown](1027)
  [pool]                cache_sanity  Cwi---C-p-   4.00g                                                       pool_cdata(0)
  [pool_cdata]          cache_sanity  Cwi-----p-   4.00g                                                       [unknown](0)
  [pool_cmeta]          cache_sanity  ewi-----p-  12.00m                                                       [unknown](1024)
  root                  rhel_host-118 -wi-ao----   6.20g                                                       /dev/vda2(205)
  swap                  rhel_host-118 -wi-ao---- 820.00m                                                       /dev/vda2(0)
[root@host-118 ~]# lvconvert --yes --uncache cache_sanity/display_cache
  WARNING: Device for PV nwSfGY-ag3Y-zQeC-C27M-Rjle-gnRt-b9KLXb not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/pool_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/pool_cmeta is missing.
  Conversion aborted.
  Cannot uncache writethrough cache volume cache_sanity/display_cache without --force.
[root@host-118 ~]# lvconvert --yes --uncache --force cache_sanity/display_cache
  WARNING: Device for PV nwSfGY-ag3Y-zQeC-C27M-Rjle-gnRt-b9KLXb not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/pool_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/pool_cmeta is missing.
  WARNING: Uncaching of partially missing writethrough cache volume cache_sanity/display_cache might destroy your data.
  /dev/mapper/cache_sanity-pool_cmeta: read failed: Input/output error
  Failed to active cache locally cache_sanity/display_cache.



Version-Release number of selected component (if applicable):
3.10.0-501.el7.x86_64

lvm2-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
lvm2-libs-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
lvm2-cluster-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-libs-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-event-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-event-libs-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016
cmirror-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
sanlock-3.4.0-1.el7    BUILT: Fri Jun 10 11:41:03 CDT 2016
sanlock-lib-3.4.0-1.el7    BUILT: Fri Jun 10 11:41:03 CDT 2016
lvm2-lockd-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016

Comment 1 Jonathan Earl Brassow 2016-09-26 19:51:09 UTC
*** Bug 1379413 has been marked as a duplicate of this bug. ***

Comment 5 Robert S. Irrgang 2018-06-08 07:40:40 UTC
I can confirm, this bug still exist!
Can't disable/remove cache, can't activate the volume.

root@vmm:~# lvs --version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync


root@vmm:~# lvchange --version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync

root@vmm:~# lvconvert --version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync


root@vmm:~# lvs -a -o -move_pv,mirror_log,copy_percent -o +devices | grep wotan
  wotan                     VM_SysDevs Cwi---C--- 16.00g [wotan_cache]       [wotan_corig]                             wotan_corig(0)
  [wotan_cache]             VM_SysDevs Cwi---C---  8.00g                                                               wotan_cache_cdata(0)
  [wotan_cache_cdata]       VM_SysDevs Cwi-------  8.00g                                                               /dev/sdj2(0)
  [wotan_cache_cmeta]       VM_SysDevs ewi------- 12.00m                                                               /dev/md127(51712)
  [wotan_corig]             VM_SysDevs owi---C--- 16.00g


root@vmm:~# lvchange -ay -v VM_SysDevs/wotan
    Activating logical volume VM_SysDevs/wotan exclusively.
    activation/volume_list configuration setting not defined: Checking only host tags for VM_SysDevs/wotan.
    Creating VM_SysDevs-wotan_cache_cdata
    Loading VM_SysDevs-wotan_cache_cdata table (253:5)
    Resuming VM_SysDevs-wotan_cache_cdata (253:5)
    Creating VM_SysDevs-wotan_cache_cmeta
    Loading VM_SysDevs-wotan_cache_cmeta table (253:7)
    Resuming VM_SysDevs-wotan_cache_cmeta (253:7)
    Creating VM_SysDevs-wotan_corig
    Loading VM_SysDevs-wotan_corig table (253:8)
    Resuming VM_SysDevs-wotan_corig (253:8)
    Executing: /usr/sbin/cache_check -q /dev/mapper/VM_SysDevs-wotan_cache_cmeta
    /usr/sbin/cache_check failed: 1
  Check of pool VM_SysDevs/wotan_cache failed (status:1). Manual repair required!
    Removing VM_SysDevs-wotan_corig (253:8)
    Removing VM_SysDevs-wotan_cache_cmeta (253:7)
    Removing VM_SysDevs-wotan_cache_cdata (253:5)


root@vmm:~# lvconvert --splitcache --force --yes --verbose VM_SysDevs/wotan
    Archiving volume group "VM_SysDevs" metadata (seqno 1015).
    activation/volume_list configuration setting not defined: Checking only host tags for VM_SysDevs/wotan.
    Creating VM_SysDevs-wotan_cache_cdata
    Loading VM_SysDevs-wotan_cache_cdata table (253:5)
    Resuming VM_SysDevs-wotan_cache_cdata (253:5)
    Creating VM_SysDevs-wotan_cache_cmeta
    Loading VM_SysDevs-wotan_cache_cmeta table (253:7)
    Resuming VM_SysDevs-wotan_cache_cmeta (253:7)
    Creating VM_SysDevs-wotan_corig
    Loading VM_SysDevs-wotan_corig table (253:8)
    Resuming VM_SysDevs-wotan_corig (253:8)
    Executing: /usr/sbin/cache_check -q /dev/mapper/VM_SysDevs-wotan_cache_cmeta
    /usr/sbin/cache_check failed: 1
  Check of pool VM_SysDevs/wotan_cache failed (status:1). Manual repair required!
    Removing VM_SysDevs-wotan_corig (253:8)
    Removing VM_SysDevs-wotan_cache_cmeta (253:7)
    Removing VM_SysDevs-wotan_cache_cdata (253:5)
  Failed to active cache locally VM_SysDevs/wotan.


root@vmm:~# lvconvert --uncache --force --yes --verbose VM_SysDevs/wotan
    Archiving volume group "VM_SysDevs" metadata (seqno 1015).
    activation/volume_list configuration setting not defined: Checking only host tags for VM_SysDevs/wotan.
    Creating VM_SysDevs-wotan_cache_cdata
    Loading VM_SysDevs-wotan_cache_cdata table (253:5)
    Resuming VM_SysDevs-wotan_cache_cdata (253:5)
    Creating VM_SysDevs-wotan_cache_cmeta
    Loading VM_SysDevs-wotan_cache_cmeta table (253:7)
    Resuming VM_SysDevs-wotan_cache_cmeta (253:7)
    Creating VM_SysDevs-wotan_corig
    Loading VM_SysDevs-wotan_corig table (253:8)
    Resuming VM_SysDevs-wotan_corig (253:8)
    Executing: /usr/sbin/cache_check -q /dev/mapper/VM_SysDevs-wotan_cache_cmeta
    /usr/sbin/cache_check failed: 1
  Check of pool VM_SysDevs/wotan_cache failed (status:1). Manual repair required!
    Removing VM_SysDevs-wotan_corig (253:8)
    Removing VM_SysDevs-wotan_cache_cmeta (253:7)
    Removing VM_SysDevs-wotan_cache_cdata (253:5)
  Failed to active cache locally VM_SysDevs/wotan.


No access to the volume in any way with bad cache device.


Note You need to log in before you can comment on or make changes to this bug.