Bug 1380532

Summary:

certain repair of cache raid volumes fails "Cannot convert internal LV "

Product:

Red Hat Enterprise Linux 7

Reporter:

Corey Marthaler <cmarthal>

Component:

lvm2

Assignee:

Heinz Mauelshagen <heinzm>

lvm2 sub component:

Cache Logical Volumes

QA Contact:

cluster-qe <cluster-qe>

Status:

CLOSED ERRATA

Docs Contact:

Milan Navratil <mnavrati>

Severity:

medium

Priority:

high

CC:

agk, heinzm, jbrassow, lmanasko, msnitzer, mthacker, pasik, prajnoha, rbednar, salmy, tlavigne, yizhan, zkabelac

Version:

7.3

Keywords:

Regression, ZStream

Target Milestone:

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

lvm2-2.02.169-1.el7

Doc Type:

Bug Fix

Doc Text:

"lvconvert --repair" now works properly on cache logical volumes Due to a regression in the lvm2-2.02.166-1.el package, released in Red Hat Enterprise Linux 7.3, the "lvconvert --repair" command could not be run properly on cache logical volumes. As a consequence, the `Cannot convert internal LV` error occurred. The underlying source code has been modified to fix this bug, and "lvconvert --repair" now works as expected.

Story Points:

---

Clone Of:

Clones:

1383925 (view as bug list)

Environment:

Last Closed:

2017-08-01 21:47:18 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1383925

Attachments:

Description	Flags
test result	none

Description Corey Marthaler 2016-09-29 21:57:43 UTC

Description of problem:
This may be expected now, or fallout from bug 1353759?

# Same failing cmd attempted manually

[root@host-123 ~]# lvconvert --repair black_bird/synced_primary_raid6_4legs_1_corig
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
  Cannot convert internal LV black_bird/synced_primary_raid6_4legs_1_corig.


================================================================================
Iteration 0.2 started at Thu Sep 29 14:50:37 CDT 2016
================================================================================
Scenario kill_primary_synced_raid6_4legs: Kill primary leg of synced 4 leg raid6 volume(s)
********* RAID hash info for this scenario *********
* names:              synced_primary_raid6_4legs_1
* sync:               1
* type:               raid6
* -m |-i value:       4
* leg devices:        /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdd1 /dev/sdg1 /dev/sdb1
* spanned legs:       0
* manual repair:      0
* failpv(s):          /dev/sde1
* failnode(s):        host-123
* lvmetad:            1
* cache stack:        1
* raid fault policy:  allocate
******************************************************

Creating raids(s) on host-123...
host-123: lvcreate  --type raid6 -i 4 -n synced_primary_raid6_4legs_1 -L 500M black_bird /dev/sde1:0-2400 /dev/sda1:0-2400 /dev/sdf1:0-2400 /dev/sdd1:0-2400 /dev/sdg1:0-2400 /dev/sdb1:0-2400

Current mirror/raid device structure(s):
  LV                                      Attr       LSize   Cpy%Sync Devices
   synced_primary_raid6_4legs_1            rwi-aor--- 512.00m 18.75    synced_primary_raid6_4legs_1_rimage_0(0),synced_primary_raid6_4legs_1_rimage_1(0),synced_primary_raid6_4legs_1_rimage_2(0),synced_primary_raid6_4legs_1_rimage_3(0),synced_primary_raid6_4legs_1_rimage_4(0),synced_primary_raid6_4legs_1_rimage_5(0)
   [synced_primary_raid6_4legs_1_rimage_0] Iwi-aor--- 128.00m          /dev/sde1(1)
   [synced_primary_raid6_4legs_1_rimage_1] Iwi-aor--- 128.00m          /dev/sda1(1)
   [synced_primary_raid6_4legs_1_rimage_2] Iwi-aor--- 128.00m          /dev/sdf1(1)
   [synced_primary_raid6_4legs_1_rimage_3] Iwi-aor--- 128.00m          /dev/sdd1(1)
   [synced_primary_raid6_4legs_1_rimage_4] Iwi-aor--- 128.00m          /dev/sdg1(1)
   [synced_primary_raid6_4legs_1_rimage_5] Iwi-aor--- 128.00m          /dev/sdb1(1)
   [synced_primary_raid6_4legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sde1(0)
   [synced_primary_raid6_4legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sda1(0)
   [synced_primary_raid6_4legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdf1(0)
   [synced_primary_raid6_4legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_primary_raid6_4legs_1_rmeta_4]  ewi-aor---   4.00m          /dev/sdg1(0)
   [synced_primary_raid6_4legs_1_rmeta_5]  ewi-aor---   4.00m          /dev/sdb1(0)


Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

********* CACHE info for this scenario *********
* Killing the raid cache ORIGIN (slow) device
* cache pool device:   /dev/sda1
* cachemode:              writethrough
****************************************************

Convert mirror/raid volume(s) to Cache volume(s) on host-123...

Creating CACHE DATA and META (fast) devices (which will not be failed), convert to cache pool create a cache volume
lvcreate -n cache_synced_primary_raid6_4legs_1 -L 500M black_bird /dev/sda1
WARNING: xfs signature detected on /dev/black_bird/cache_synced_primary_raid6_4legs_1 at offset 0. Wipe it? [y/n]: [n]
  Aborted wiping of xfs.
  1 existing signature left on the device.
lvcreate -n meta -L 12M black_bird /dev/sda1

Create cache pool volume by combining the cache data and cache metadata (fast) volumes
lvconvert --yes --type cache-pool --cachemode writethrough --poolmetadata black_bird/meta black_bird/cache_synced_primary_raid6_4legs_1
  WARNING: Converting logical volume black_bird/cache_synced_primary_raid6_4legs_1 and black_bird/meta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Create cached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type cache --cachepool black_bird/cache_synced_primary_raid6_4legs_1 black_bird/synced_primary_raid6_4legs_1

Creating xfs on top of mirror(s) on host-123...
Mounting mirrored xfs filesystems on host-123...

Current mirror/raid device structure(s):
  LV                                            Attr       LSize   Cpy%Sync Devices
   [cache_synced_primary_raid6_4legs_1]          Cwi---C--- 500.00m 0.00     cache_synced_primary_raid6_4legs_1_cdata(0)
   [cache_synced_primary_raid6_4legs_1_cdata]    Cwi-ao---- 500.00m          /dev/sda1(33)
   [cache_synced_primary_raid6_4legs_1_cmeta]    ewi-ao----  12.00m          /dev/sda1(158)
   [lvol0_pmspare]                               ewi-------  12.00m          /dev/sde1(33)
   synced_primary_raid6_4legs_1                  Cwi-aoC--- 512.00m 0.00     synced_primary_raid6_4legs_1_corig(0)
   [synced_primary_raid6_4legs_1_corig]          rwi-aoC--- 512.00m 100.00   synced_primary_raid6_4legs_1_corig_rimage_0(0),synced_primary_raid6_4legs_1_corig_rimage_1(0),synced_primary_raid6_4legs_1_corig_rimage_2(0),synced_primary_raid6_4legs_1_corig_rimage_3(0),synced_primary_raid6_4legs_1_corig_rimage_4(0),synced_primary_raid6_4legs_1_corig_rimage_5(0)
   [synced_primary_raid6_4legs_1_corig_rimage_0] iwi-aor--- 128.00m          /dev/sde1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_1] iwi-aor--- 128.00m          /dev/sda1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_2] iwi-aor--- 128.00m          /dev/sdf1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_3] iwi-aor--- 128.00m          /dev/sdd1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_4] iwi-aor--- 128.00m          /dev/sdg1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_5] iwi-aor--- 128.00m          /dev/sdb1(1)
   [synced_primary_raid6_4legs_1_corig_rmeta_0]  ewi-aor---   4.00m          /dev/sde1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_1]  ewi-aor---   4.00m          /dev/sda1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_2]  ewi-aor---   4.00m          /dev/sdf1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_3]  ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_4]  ewi-aor---   4.00m          /dev/sdg1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_5]  ewi-aor---   4.00m          /dev/sdb1(0)

PV=/dev/sde1
        lvol0_pmspare: 1.0
        synced_primary_raid6_4legs_1_corig_rimage_0: 1.0
        synced_primary_raid6_4legs_1_corig_rmeta_0: 1.0

Writing verification files (checkit) to mirror(s) on...
        ---- host-123 ----

<start name="host-123_synced_primary_raid6_4legs_1"  pid="7258" time="Thu Sep 29 14:51:18 2016" type="cmd" />
Sleeping 15 seconds to get some outsanding I/O locks before the failure 

Verifying files (checkit) on mirror(s) on...
        ---- host-123 ----



Disabling device sde on host-123rescan device...
  /dev/sde1: read failed after 0 of 512 at 21467824128: Input/output error
  /dev/sde1: read failed after 0 of 512 at 21467938816: Input/output error
  /dev/sde1: read failed after 0 of 512 at 0: Input/output error
  /dev/sde1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sde1: read failed after 0 of 2048 at 0: Input/output error


Getting recovery check start time from /var/log/messages: Sep 29 14:51
Attempting I/O to cause mirror down conversion(s) on host-123
dd if=/dev/zero of=/mnt/synced_primary_raid6_4legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.173477 s, 242 MB/s

Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
  LV                                            Attr       LSize   Cpy%Sync Devices
   [cache_synced_primary_raid6_4legs_1]          Cwi---C--- 500.00m 0.00     cache_synced_primary_raid6_4legs_1_cdata(0)
   [cache_synced_primary_raid6_4legs_1_cdata]    Cwi-ao---- 500.00m          /dev/sda1(33)
   [cache_synced_primary_raid6_4legs_1_cmeta]    ewi-ao----  12.00m          /dev/sda1(158)
   [lvol0_pmspare]                               ewi-----p-  12.00m          [unknown](33)
   synced_primary_raid6_4legs_1                  Cwi-aoC-p- 512.00m 0.00     synced_primary_raid6_4legs_1_corig(0)
   [synced_primary_raid6_4legs_1_corig]          rwi-aoC-p- 512.00m 100.00   synced_primary_raid6_4legs_1_corig_rimage_0(0),synced_primary_raid6_4legs_1_corig_rimage_1(0),synced_primary_raid6_4legs_1_corig_rimage_2(0),synced_primary_raid6_4legs_1_corig_rimage_3(0),synced_primary_raid6_4legs_1_corig_rimage_4(0),synced_primary_raid6_4legs_1_corig_rimage_5(0)
   [synced_primary_raid6_4legs_1_corig_rimage_0] iwi-aor-p- 128.00m          [unknown](1)
   [synced_primary_raid6_4legs_1_corig_rimage_1] iwi-aor--- 128.00m          /dev/sda1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_2] iwi-aor--- 128.00m          /dev/sdf1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_3] iwi-aor--- 128.00m          /dev/sdd1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_4] iwi-aor--- 128.00m          /dev/sdg1(1)
   [synced_primary_raid6_4legs_1_corig_rimage_5] iwi-aor--- 128.00m          /dev/sdb1(1)
   [synced_primary_raid6_4legs_1_corig_rmeta_0]  ewi-aor-p-   4.00m          [unknown](0)
   [synced_primary_raid6_4legs_1_corig_rmeta_1]  ewi-aor---   4.00m          /dev/sda1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_2]  ewi-aor---   4.00m          /dev/sdf1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_3]  ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_4]  ewi-aor---   4.00m          /dev/sdg1(0)
   [synced_primary_raid6_4legs_1_corig_rmeta_5]  ewi-aor---   4.00m          /dev/sdb1(0)

Verifying FAILED device /dev/sde1 is *NOT* in the volume(s)
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Verifying IMAGE device /dev/sda1 *IS* in the volume(s)
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Verifying IMAGE device /dev/sdf1 *IS* in the volume(s)
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Verifying IMAGE device /dev/sdd1 *IS* in the volume(s)
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Verifying IMAGE device /dev/sdg1 *IS* in the volume(s)
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
  WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of lvol0_pmspare on: 
        Skipping check for lvol0_pmspare until BUG 1030121 (lvol0_pmspare) is sorted out
Checking EXISTENCE and STATE of synced_primary_raid6_4legs_1_corig_rimage_0 on: host-123 

(ALLOCATE POLICY) there should not be an 'unknown' device associated with synced_primary_raid6_4legs_1_corig_rimage_0 on host-123



Sep 29 14:51:39 host-123 qarshd[18024]: Running cmdline: echo offline > /sys/block/sde/device/state
Sep 29 14:51:39 host-123 systemd: Started qarsh Per-Connection Server (10.15.80.224:50172).
Sep 29 14:51:39 host-123 systemd: Starting qarsh Per-Connection Server (10.15.80.224:50172)...
Sep 29 14:51:39 host-123 qarshd[18028]: Talking to peer ::ffff:10.15.80.224:50172 (IPv6)
Sep 29 14:51:39 host-123 qarshd[18028]: Running cmdline: pvscan --cache /dev/sde1
Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device
Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device
Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device
Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device
Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device
Sep 29 14:51:43 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device
Sep 29 14:51:43 host-123 kernel: md: super_written gets error=-5, uptodate=0
Sep 29 14:51:43 host-123 kernel: md/raid:mdX: Disk failure on dm-3, disabling device.#012md/raid:mdX: Operation continuing on 5 devices.
Sep 29 14:51:43 host-123 lvm[16364]: Device #0 of raid6_zr array, black_bird-synced_primary_raid6_4legs_1_corig, has failed.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr already missing, skipping.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices.
Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices.
Sep 29 14:51:43 host-123 lvm[16364]: Cannot convert internal LV black_bird/synced_primary_raid6_4legs_1_corig.
Sep 29 14:51:43 host-123 lvm[16364]: Failed to process event for black_bird-synced_primary_raid6_4legs_1_corig.


Version-Release number of selected component (if applicable):
3.10.0-510.el7.x86_64

lvm2-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
lvm2-libs-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
lvm2-cluster-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-libs-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-event-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-event-libs-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016

Comment 1 Corey Marthaler 2016-09-29 22:00:54 UTC

This appear to only affect cache raid repair when the fault policy is "allocate", warn with a manual repair attempt appears to succeed.

Comment 3 Heinz Mauelshagen 2016-10-04 15:21:40 UTC

Reproduced with raid_fault_policy="allocate".

"lvconvert --repair black_bird/synced_primary_raid6_4legs_1_corig" fails here with "Cannot convert internal LV ...".

Is this a regression at all?

Comment 4 Corey Marthaler 2016-10-04 16:24:45 UTC

1. verified that a manual repair is never attempted due to the restriction that cache and pool raids need to be inactive during the passing 'warn' cases, so disregard comment #1.


2. verified that this same test case passed in 7.2.z as well as in a previous version of rhel7.3:

3.10.0-510.el7.x86_64

lvm2-2.02.161-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
lvm2-libs-2.02.161-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
lvm2-cluster-2.02.161-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
device-mapper-1.02.131-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
device-mapper-libs-1.02.131-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
device-mapper-event-1.02.131-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
device-mapper-event-libs-1.02.131-3.el7    BUILT: Thu Jul 28 09:31:24 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016



Oct  4 11:17:01 host-116 lvm[3815]: Device #0 of raid1 array, black_bird-synced_primary_raid1_2legs_1_cdata, has failed.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rimage_0 while checking used and assumed devices.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rmeta_0 while checking used and assumed devices.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V already missing, skipping.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rimage_0 while checking used and assumed devices.
Oct  4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rmeta_0 while checking used and assumed devices.
Oct  4 11:17:02 host-116 kernel: device-mapper: raid: Device 0 specified for rebuild; clearing superblock
Oct  4 11:17:02 host-116 kernel: md/raid1:mdX: active with 1 out of 2 mirrors
Oct  4 11:17:02 host-116 kernel: created bitmap (1 pages) for device mdX
Oct  4 11:17:03 host-116 kernel: mdX: bitmap initialized from disk: read 1 pages, set 0 of 1000 bits
Oct  4 11:17:03 host-116 kernel: md: recovery of RAID array mdX
Oct  4 11:17:03 host-116 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Oct  4 11:17:03 host-116 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Oct  4 11:17:03 host-116 kernel: md: using 128k window, over a total of 512000k.
Oct  4 11:17:03 host-116 multipathd: dm-2: remove map (uevent)
Oct  4 11:17:03 host-116 multipathd: dm-2: devmap not registered, can't remove
Oct  4 11:17:03 host-116 multipathd: dm-3: remove map (uevent)
Oct  4 11:17:03 host-116 multipathd: dm-3: devmap not registered, can't remove
Oct  4 11:17:03 host-116 multipathd: dm-13: remove map (uevent)
Oct  4 11:17:03 host-116 multipathd: dm-13: devmap not registered, can't remove
Oct  4 11:17:03 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51342).
Oct  4 11:17:03 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51342)...
Oct  4 11:17:03 host-116 multipathd: dm-12: remove map (uevent)
Oct  4 11:17:03 host-116 multipathd: dm-12: devmap not registered, can't remove
Oct  4 11:17:03 host-116 multipathd: dm-2: remove map (uevent)
Oct  4 11:17:03 host-116 multipathd: dm-3: remove map (uevent)
Oct  4 11:17:03 host-116 qarshd[6698]: Talking to peer ::ffff:10.15.80.224:51342 (IPv6)
Oct  4 11:17:03 host-116 multipathd: dm-12: remove map (uevent)
Oct  4 11:17:03 host-116 multipathd: dm-13: remove map (uevent)
Oct  4 11:17:03 host-116 qarshd[6698]: Running cmdline: pvs -a
Oct  4 11:17:03 host-116 kernel: md/raid1:mdX: active with 1 out of 2 mirrors
Oct  4 11:17:03 host-116 kernel: created bitmap (1 pages) for device mdX
Oct  4 11:17:04 host-116 kernel: md: mdX: recovery interrupted.
Oct  4 11:17:04 host-116 kernel: mdX: bitmap initialized from disk: read 1 pages, set 0 of 1000 bits
Oct  4 11:17:04 host-116 kernel: md: recovery of RAID array mdX
Oct  4 11:17:04 host-116 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Oct  4 11:17:04 host-116 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Oct  4 11:17:04 host-116 kernel: md: using 128k window, over a total of 512000k.
Oct  4 11:17:04 host-116 kernel: md: resuming recovery of mdX from checkpoint.
Oct  4 11:17:04 host-116 lvm[3815]: Faulty devices in black_bird/synced_primary_raid1_2legs_1_cdata successfully replaced.
Oct  4 11:17:04 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51344).
Oct  4 11:17:04 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51344)...
Oct  4 11:17:04 host-116 qarshd[6751]: Talking to peer ::ffff:10.15.80.224:51344 (IPv6)
Oct  4 11:17:04 host-116 qarshd[6751]: Running cmdline: tail -n1 /var/log/messages | cut -c 1-12
Oct  4 11:17:04 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51346).
Oct  4 11:17:04 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51346)...
Oct  4 11:17:04 host-116 qarshd[6757]: Talking to peer ::ffff:10.15.80.224:51346 (IPv6)
Oct  4 11:17:05 host-116 qarshd[6757]: Running cmdline: dd if=/dev/zero of=/mnt/synced_primary_raid1_2legs_1/ddfile count=10 bs=4M
Oct  4 11:17:05 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51348).
Oct  4 11:17:05 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51348)...
Oct  4 11:17:05 host-116 qarshd[6761]: Talking to peer ::ffff:10.15.80.224:51348 (IPv6)
Oct  4 11:17:05 host-116 qarshd[6761]: Running cmdline: sync
Oct  4 11:17:08 host-116 kernel: md: mdX: recovery done.
Oct  4 11:17:08 host-116 lvm[3815]: Device #0 of raid1 array, black_bird-synced_primary_raid1_2legs_1_cdata, has failed.
Oct  4 11:17:08 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter.
Oct  4 11:17:08 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter.
Oct  4 11:17:08 host-116 lvm[3815]: Faulty devices in black_bird/synced_primary_raid1_2legs_1_cdata successfully replaced.

Comment 6 Jonathan Earl Brassow 2016-10-04 17:16:18 UTC

There is a work-around for this issue (I'll elaborate on that in next comment).  It should not cause hangs/corruptions - only inconvenience.  As such, I am pushing for 7.4 fix with 7.3.z inclusion.

Comment 7 Heinz Mauelshagen 2016-10-05 16:03:36 UTC

Got one-liner fix to lvconvert.c allowing for the rebuild of corig internal LV in testing...

Comment 8 Heinz Mauelshagen 2016-10-10 15:33:00 UTC

Fix pushed upstream:

allow repair on cache orig RAID LVs and
"lvconvert --replace/--mirrors/--type {raid*|mirror|striped|linear}" as well.
    
Allow the same lvconvert actions on any cache pool and metadata RAID SubLVs.

Comment 18 Roman Bednář 2017-04-18 12:02:47 UTC

Created attachment 1272293 [details]
test result

Thank you for explanation, I was not aware of this behaviour.

Marking verified.

3.10.0-640.el7.x86_64

lvm2-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
lvm2-libs-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
lvm2-cluster-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-libs-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-event-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-event-libs-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 17:15:46 CEST 2017
cmirror-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017

Comment 19 errata-xmlrpc 2017-08-01 21:47:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2222