1377850 – LVM cache: allow abandoning cachepool (lvconvert --uncache) if it has failed, remaining scenarios

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1377850 - LVM cache: allow abandoning cachepool (lvconvert --uncache) if it has failed, remaining scenarios

Summary: LVM cache: allow abandoning cachepool (lvconvert --uncache) if it has failed,...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.3
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Zdenek Kabelac
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1379413 (view as bug list)
Depends On:	1379472
Blocks:
TreeView+	depends on / blocked

Reported:	2016-09-20 19:01 UTC by Corey Marthaler
Modified:	2021-09-03 12:56 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-12-15 07:46:10 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Corey Marthaler 2016-09-20 19:01:57 UTC

Description of problem:
This bug is for the remaining cache pool failure issues found in https://bugzilla.redhat.com/show_bug.cgi?id=1131777#c11


# scenario 1: fail the cache pool device, attempt an uncache: FAILS
[root@host-117 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sda1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sda1(1024)       

[root@host-117 ~]# echo offline > /sys/block/sda/device/state
[root@host-117 ~]# lvconvert --uncache cache_sanity/display_cache
  /dev/sda1: read failed after 0 of 512 at 4096: Input/output error
  WARNING: Failed to write an MDA of VG cache_sanity.
[DEADLOCK]


# scenario 2: fail the cache pool device, attempt a pvscan: FAILS
[root@host-116 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sdd1(1024)       
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdf1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdf1(1024)       

[root@host-116 ~]# echo offline > /sys/block/sdf/device/state
[root@host-116 ~]# pvscan --cache
[DEADLOCK]


# scenario 3: fail the cache pool device, attempt a splitcache, then attempt an uncache: PASSES
[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   8.66   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sdf1(1024)       
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   8.66   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdd1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdd1(1024)       
[root@host-118 ~]# lvconvert --yes --splitcache cache_sanity/display_cache
  /dev/sdd1: read failed after 0 of 512 at 4096: Input/output error
  Failed to write VG cache_sanity.

[441927.867015] sd 4:0:0:1: rejecting I/O to offline device
[441927.867015] sd 4:0:0:1: rejecting I/O to offline device
[441927.874658] metadata_operation_failed: 384 callbacks suppressed
[441927.876002] device-mapper: cache: 253:2: metadata operation 'dm_cache_commit' failed: error = -5
[441927.877705] device-mapper: cache: 253:2: aborting current metadata transaction
[441927.879073] sd 4:0:0:1: rejecting I/O to offline device
[441927.881134] sd 4:0:0:1: rejecting I/O to offline device
[441927.881774] device-mapper: cache: 253:2: failed to abort metadata transaction
[441927.882603] device-mapper: cache: unable to read needs_check flag, setting failure mode
[441927.883701] device-mapper: cache: 253:2: switching cache to fail mode
[441927.884599] device-mapper: cache: unable to read needs_check flag, setting failure mode
[441927.885918] device-mapper: cache: 253:2: could not write cache metadata
[441927.899748] sd 4:0:0:1: rejecting I/O to offline device

[root@host-118 ~]# lvconvert --yes --uncache cache_sanity/display_cache
  WARNING: Skippping flush for failed cache cache_sanity/display_cache.
  /dev/sdd1: read failed after 0 of 512 at 4096: Input/output error
  WARNING: Failed to write an MDA of VG cache_sanity.
  WARNING: Failed to write an MDA of VG cache_sanity.
  Logical volume "pool" successfully removed
  Logical volume cache_sanity/display_cache is not cached.
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV vFLT6Z-sj38-ivEh-lC3n-VkWV-5V6t-OkUYZD not found or rejected by a filter.
  LV            VG            Attr       LSize   Pool Origin Data%  Meta%  Cpy%Sync Devices       
  display_cache cache_sanity  -wi-a-----   4.00g                                    /dev/sdf1(0)  


# scenario 4: fail the cache pool device, attempt a write to the cache origin: FAILS
[root@host-117 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdd1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sda1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sda1(1024)       
[root@host-117 ~]#  echo offline > /sys/block/sda/device/state
[root@host-117 ~]# dd if=/dev/zero of=/dev/cache_sanity/display_cache count=1 bs=1M
[DEADLOCK]



# scenario 5: corrupt the cache pool device by writing zeros over it, attempt an uncache: FAILS
[root@host-116 ~]# lvs -a -o +devices
  WARNING: Device for PV ZFKtOZ-eSBT-eAdB-PBsD-hWOs-sqRL-0TcP3K not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi-a-C-p-   4.00g [corrupt_pool_meta]        0.00   2.31   0.00     corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi-aoC---   4.00g                                                   /dev/sda1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                            0.00   2.31   0.00     corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-ao--p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-ao--p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sdd1(0)              

[root@host-116 ~]# lvconvert --uncache cache_sanity/corigin
  WARNING: Device for PV ZFKtOZ-eSBT-eAdB-PBsD-hWOs-sqRL-0TcP3K not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  WARNING: Cache pool data logical volume cache_sanity/corrupt_pool_meta_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/corrupt_pool_meta_cmeta is missing.
Do you really want to uncache cache_sanity/corigin with missing LVs? [y/n]: y
[DEADLOCK]

Sep 19 15:11:59 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: migration_failure: 7719 callbacks suppressed
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:04 host-116 kernel: device-mapper: cache: 253:2: promotion failed; couldn't copy block
Sep 19 15:12:05 host-116 kernel: INFO: task systemd-udevd:3928 blocked for more than 120 seconds.
Sep 19 15:12:05 host-116 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 19 15:12:05 host-116 kernel: systemd-udevd   D ffff88003b993a18     0  3928    476 0x00000086
Sep 19 15:12:05 host-116 kernel: ffff88003b9938b0 0000000000000046 ffff88003b54edd0 ffff88003b993fd8
Sep 19 15:12:05 host-116 kernel: ffff88003b993fd8 ffff88003b993fd8 ffff88003b54edd0 ffff88003fc16c40
Sep 19 15:12:05 host-116 kernel: 0000000000000000 7fffffffffffffff ffffffff81686b20 ffff88003b993a18
Sep 19 15:12:05 host-116 kernel: Call Trace:
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b20>] ? bit_wait+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff81688aa9>] schedule+0x29/0x70
Sep 19 15:12:05 host-116 kernel: [<ffffffff816864f9>] schedule_timeout+0x239/0x2d0
Sep 19 15:12:05 host-116 kernel: [<ffffffffa03722f0>] ? cache_resume+0x30/0x30 [dm_cache]
Sep 19 15:12:05 host-116 kernel: [<ffffffff81060aef>] ? kvm_clock_get_cycles+0x1f/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b20>] ? bit_wait+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff8168804e>] io_schedule_timeout+0xae/0x130
Sep 19 15:12:05 host-116 kernel: [<ffffffff816880e8>] io_schedule+0x18/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff81686b31>] bit_wait_io+0x11/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff8168684f>] __wait_on_bit_lock+0x5f/0xc0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8117f944>] __lock_page+0x74/0x90
Sep 19 15:12:05 host-116 kernel: [<ffffffff810b1590>] ? wake_bit_function+0x40/0x40
Sep 19 15:12:05 host-116 kernel: [<ffffffff811908ee>] truncate_inode_pages_range+0x6fe/0x740
Sep 19 15:12:05 host-116 kernel: [<ffffffff8118dca1>] ? pagevec_lookup_tag+0x21/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff813170b5>] ? cpumask_next_and+0x35/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff810f943a>] ? smp_call_function_many+0x5a/0x260
Sep 19 15:12:05 host-116 kernel: [<ffffffff81230f40>] ? mark_buffer_async_write+0x20/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff81317199>] ? free_cpumask_var+0x9/0x10
Sep 19 15:12:05 host-116 kernel: [<ffffffff810f982d>] ? on_each_cpu_cond+0xcd/0x180
Sep 19 15:12:05 host-116 kernel: [<ffffffff81231d70>] ? __brelse+0x50/0x50
Sep 19 15:12:05 host-116 kernel: [<ffffffff81190945>] truncate_inode_pages+0x15/0x20
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123891f>] kill_bdev+0x2f/0x40
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123a1e4>] __blkdev_put+0x64/0x1a0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123ac7e>] blkdev_put+0x4e/0x140
Sep 19 15:12:05 host-116 kernel: [<ffffffff8123ae25>] blkdev_close+0x25/0x30
Sep 19 15:12:05 host-116 kernel: [<ffffffff811ff7e9>] __fput+0xe9/0x260
Sep 19 15:12:05 host-116 kernel: [<ffffffff811ffa9e>] ____fput+0xe/0x10
Sep 19 15:12:05 host-116 kernel: [<ffffffff810acbb4>] task_work_run+0xc4/0xe0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8108bb38>] do_exit+0x2d8/0xa40
Sep 19 15:12:05 host-116 kernel: [<ffffffff812193cf>] ? touch_atime+0x12f/0x160
Sep 19 15:12:05 host-116 kernel: [<ffffffff81181fb8>] ? generic_file_aio_read+0x5d8/0x790
Sep 19 15:12:05 host-116 kernel: [<ffffffff8108c31f>] do_group_exit+0x3f/0xa0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8109d390>] get_signal_to_deliver+0x1d0/0x6d0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8102a427>] do_signal+0x57/0x6c0
Sep 19 15:12:05 host-116 kernel: [<ffffffff8102aaef>] do_notify_resume+0x5f/0xb0



# scenario 6: corrupt the cache pool device by writing zeros over it, reactivate in partial mode, attempt an uncache: PASSES
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi-a-C-p-   4.00g [corrupt_pool_meta]        0.00   4.39   0.00     corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi-aoC---   4.00g                                                   /dev/sdf1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                            0.00   4.39   0.00     corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-ao--p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-ao--p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sde1(0)              

[root@host-118 ~]# vgchange -an --partial cache_sanity
  PARTIAL MODE. Incomplete logical volumes will be processed.
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cdata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV cache_sanity/corrupt_pool_meta_cmeta while checking used and assumed devices.
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  LV                        VG            Attr       LSize   Pool                Origin Data%  Meta%  Cpy%Sync Devices                   
  corigin                   cache_sanity  Cwi---C-p-   4.00g [corrupt_pool_meta]                               corigin_corig(0)          
  [corigin_corig]           cache_sanity  owi---C---   4.00g                                                   /dev/sdf1(0)              
  [corrupt_pool_meta]       cache_sanity  Cwi---C-p-   2.00g                                                   corrupt_pool_meta_cdata(0)
  [corrupt_pool_meta_cdata] cache_sanity  Cwi-----p-   2.00g                                                   [unknown](0)              
  [corrupt_pool_meta_cmeta] cache_sanity  ewi-----p-  12.00m                                                   [unknown](512)            
  [lvol0_pmspare]           cache_sanity  ewi-------  12.00m                                                   /dev/sde1(0)              
[root@host-118 ~]# lvconvert --uncache cache_sanity/corigin
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/corrupt_pool_meta_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/corrupt_pool_meta_cmeta is missing.
Do you really want to uncache cache_sanity/corigin with missing LVs? [y/n]: y
  Logical volume "corrupt_pool_meta" successfully removed
  Logical volume cache_sanity/corigin is not cached.
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV LbJ365-Hca3-YwIk-cWVw-22Ei-DcJ0-TN0zUL not found or rejected by a filter.
  LV      VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  corigin cache_sanity  -wi-------   4.00g                                                     /dev/sdf1(0)  


# scenario 7: fail the cache pool device, attempt to reactivate in partial mode, attempt an uncache: FAILS
[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi-a-C---   4.00g [pool]        0.00   4.39   0.00     display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi-aoC---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g               0.00   4.39   0.00     pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-ao----   4.00g                                      /dev/sdc1(0)          
  [pool_cmeta]          cache_sanity  ewi-ao----  12.00m                                      /dev/sdc1(1024)       

[root@host-118 ~]# echo offline > /sys/block/sdc/device/state
[root@host-118 ~]#  vgchange -an --partial cache_sanity
  PARTIAL MODE. Incomplete logical volumes will be processed.
  /dev/mapper/cache_sanity-pool_cmeta: read failed: Input/output error
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  /dev/sdc1: open failed: No such device or address
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Cpy%Sync Devices               
  display_cache         cache_sanity  Cwi---C---   4.00g [pool]                               display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                      /dev/sdf1(0)          
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                      /dev/sde1(0)          
  [pool]                cache_sanity  Cwi---C---   4.00g                                      pool_cdata(0)         
  [pool_cdata]          cache_sanity  Cwi-------   4.00g                                      /dev/sdc1(0)          
  [pool_cmeta]          cache_sanity  ewi-------  12.00m                                      /dev/sdc1(1024)       

[root@host-118 ~]# lvconvert --uncache cache_sanity/display_cache
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  /dev/sdc1: open failed: No such device or address
  device-mapper: reload ioctl on (253:2) failed: No such device or address
  Failed to active cache locally cache_sanity/display_cache.
  Failed to uncache cache_sanity/display_cache.


# Additional scenario 8: the pmspare device is the same PV as the failed cache pool.
[root@host-118 ~]# vgchange -an cache_sanity
  0 logical volume(s) in volume group "cache_sanity" now active

[root@host-118 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  display_cache         cache_sanity  Cwi---C---   4.00g [pool]                                                display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                                       /dev/sdd1(0)   
  [lvol0_pmspare]       cache_sanity  ewi-------  12.00m                                                       /dev/sda1(1027)
  [pool]                cache_sanity  Cwi---C---   4.00g                                                       pool_cdata(0)
  [pool_cdata]          cache_sanity  Cwi-------   4.00g                                                       /dev/sda1(0)
  [pool_cmeta]          cache_sanity  ewi-------  12.00m                                                       /dev/sda1(1024)

[root@host-118 ~]# echo offline > /sys/block/sda/device/state
[root@host-118 ~]# pvscan --cache
[root@host-118 ~]# lvs -a -o +devices
  WARNING: Device for PV nwSfGY-ag3Y-zQeC-C27M-Rjle-gnRt-b9KLXb not found or rejected by a filter.
  LV                    VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  display_cache         cache_sanity  Cwi---C-p-   4.00g [pool]                                                display_cache_corig(0)
  [display_cache_corig] cache_sanity  owi---C---   4.00g                                                       /dev/sdd1(0)
  [lvol0_pmspare]       cache_sanity  ewi-----p-  12.00m                                                       [unknown](1027)
  [pool]                cache_sanity  Cwi---C-p-   4.00g                                                       pool_cdata(0)
  [pool_cdata]          cache_sanity  Cwi-----p-   4.00g                                                       [unknown](0)
  [pool_cmeta]          cache_sanity  ewi-----p-  12.00m                                                       [unknown](1024)
  root                  rhel_host-118 -wi-ao----   6.20g                                                       /dev/vda2(205)
  swap                  rhel_host-118 -wi-ao---- 820.00m                                                       /dev/vda2(0)
[root@host-118 ~]# lvconvert --yes --uncache cache_sanity/display_cache
  WARNING: Device for PV nwSfGY-ag3Y-zQeC-C27M-Rjle-gnRt-b9KLXb not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/pool_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/pool_cmeta is missing.
  Conversion aborted.
  Cannot uncache writethrough cache volume cache_sanity/display_cache without --force.
[root@host-118 ~]# lvconvert --yes --uncache --force cache_sanity/display_cache
  WARNING: Device for PV nwSfGY-ag3Y-zQeC-C27M-Rjle-gnRt-b9KLXb not found or rejected by a filter.
  WARNING: Cache pool data logical volume cache_sanity/pool_cdata is missing.
  WARNING: Cache pool metadata logical volume cache_sanity/pool_cmeta is missing.
  WARNING: Uncaching of partially missing writethrough cache volume cache_sanity/display_cache might destroy your data.
  /dev/mapper/cache_sanity-pool_cmeta: read failed: Input/output error
  Failed to active cache locally cache_sanity/display_cache.



Version-Release number of selected component (if applicable):
3.10.0-501.el7.x86_64

lvm2-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
lvm2-libs-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
lvm2-cluster-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-libs-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-event-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-event-libs-1.02.134-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016
cmirror-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016
sanlock-3.4.0-1.el7    BUILT: Fri Jun 10 11:41:03 CDT 2016
sanlock-lib-3.4.0-1.el7    BUILT: Fri Jun 10 11:41:03 CDT 2016
lvm2-lockd-2.02.165-2.el7    BUILT: Wed Sep 14 09:01:43 CDT 2016

Comment 1 Jonathan Earl Brassow 2016-09-26 19:51:09 UTC

*** Bug 1379413 has been marked as a duplicate of this bug. ***

Comment 5 Robert S. Irrgang 2018-06-08 07:40:40 UTC

I can confirm, this bug still exist!
Can't disable/remove cache, can't activate the volume.

root@vmm:~# lvs --version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync


root@vmm:~# lvchange --version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync

root@vmm:~# lvconvert --version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync


root@vmm:~# lvs -a -o -move_pv,mirror_log,copy_percent -o +devices | grep wotan
  wotan                     VM_SysDevs Cwi---C--- 16.00g [wotan_cache]       [wotan_corig]                             wotan_corig(0)
  [wotan_cache]             VM_SysDevs Cwi---C---  8.00g                                                               wotan_cache_cdata(0)
  [wotan_cache_cdata]       VM_SysDevs Cwi-------  8.00g                                                               /dev/sdj2(0)
  [wotan_cache_cmeta]       VM_SysDevs ewi------- 12.00m                                                               /dev/md127(51712)
  [wotan_corig]             VM_SysDevs owi---C--- 16.00g


root@vmm:~# lvchange -ay -v VM_SysDevs/wotan
    Activating logical volume VM_SysDevs/wotan exclusively.
    activation/volume_list configuration setting not defined: Checking only host tags for VM_SysDevs/wotan.
    Creating VM_SysDevs-wotan_cache_cdata
    Loading VM_SysDevs-wotan_cache_cdata table (253:5)
    Resuming VM_SysDevs-wotan_cache_cdata (253:5)
    Creating VM_SysDevs-wotan_cache_cmeta
    Loading VM_SysDevs-wotan_cache_cmeta table (253:7)
    Resuming VM_SysDevs-wotan_cache_cmeta (253:7)
    Creating VM_SysDevs-wotan_corig
    Loading VM_SysDevs-wotan_corig table (253:8)
    Resuming VM_SysDevs-wotan_corig (253:8)
    Executing: /usr/sbin/cache_check -q /dev/mapper/VM_SysDevs-wotan_cache_cmeta
    /usr/sbin/cache_check failed: 1
  Check of pool VM_SysDevs/wotan_cache failed (status:1). Manual repair required!
    Removing VM_SysDevs-wotan_corig (253:8)
    Removing VM_SysDevs-wotan_cache_cmeta (253:7)
    Removing VM_SysDevs-wotan_cache_cdata (253:5)


root@vmm:~# lvconvert --splitcache --force --yes --verbose VM_SysDevs/wotan
    Archiving volume group "VM_SysDevs" metadata (seqno 1015).
    activation/volume_list configuration setting not defined: Checking only host tags for VM_SysDevs/wotan.
    Creating VM_SysDevs-wotan_cache_cdata
    Loading VM_SysDevs-wotan_cache_cdata table (253:5)
    Resuming VM_SysDevs-wotan_cache_cdata (253:5)
    Creating VM_SysDevs-wotan_cache_cmeta
    Loading VM_SysDevs-wotan_cache_cmeta table (253:7)
    Resuming VM_SysDevs-wotan_cache_cmeta (253:7)
    Creating VM_SysDevs-wotan_corig
    Loading VM_SysDevs-wotan_corig table (253:8)
    Resuming VM_SysDevs-wotan_corig (253:8)
    Executing: /usr/sbin/cache_check -q /dev/mapper/VM_SysDevs-wotan_cache_cmeta
    /usr/sbin/cache_check failed: 1
  Check of pool VM_SysDevs/wotan_cache failed (status:1). Manual repair required!
    Removing VM_SysDevs-wotan_corig (253:8)
    Removing VM_SysDevs-wotan_cache_cmeta (253:7)
    Removing VM_SysDevs-wotan_cache_cdata (253:5)
  Failed to active cache locally VM_SysDevs/wotan.


root@vmm:~# lvconvert --uncache --force --yes --verbose VM_SysDevs/wotan
    Archiving volume group "VM_SysDevs" metadata (seqno 1015).
    activation/volume_list configuration setting not defined: Checking only host tags for VM_SysDevs/wotan.
    Creating VM_SysDevs-wotan_cache_cdata
    Loading VM_SysDevs-wotan_cache_cdata table (253:5)
    Resuming VM_SysDevs-wotan_cache_cdata (253:5)
    Creating VM_SysDevs-wotan_cache_cmeta
    Loading VM_SysDevs-wotan_cache_cmeta table (253:7)
    Resuming VM_SysDevs-wotan_cache_cmeta (253:7)
    Creating VM_SysDevs-wotan_corig
    Loading VM_SysDevs-wotan_corig table (253:8)
    Resuming VM_SysDevs-wotan_corig (253:8)
    Executing: /usr/sbin/cache_check -q /dev/mapper/VM_SysDevs-wotan_cache_cmeta
    /usr/sbin/cache_check failed: 1
  Check of pool VM_SysDevs/wotan_cache failed (status:1). Manual repair required!
    Removing VM_SysDevs-wotan_corig (253:8)
    Removing VM_SysDevs-wotan_cache_cmeta (253:7)
    Removing VM_SysDevs-wotan_cache_cdata (253:5)
  Failed to active cache locally VM_SysDevs/wotan.


No access to the volume in any way with bad cache device.

Comment 7 RHEL Program Management 2020-12-15 07:46:10 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Note You need to log in before you can comment on or make changes to this bug.