Red Hat Bugzilla – Bug 1380532
certain repair of cache raid volumes fails "Cannot convert internal LV "
Last modified: 2017-08-02 04:13:07 EDT
Description of problem: This may be expected now, or fallout from bug 1353759? # Same failing cmd attempted manually [root@host-123 ~]# lvconvert --repair black_bird/synced_primary_raid6_4legs_1_corig WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Cannot convert internal LV black_bird/synced_primary_raid6_4legs_1_corig. ================================================================================ Iteration 0.2 started at Thu Sep 29 14:50:37 CDT 2016 ================================================================================ Scenario kill_primary_synced_raid6_4legs: Kill primary leg of synced 4 leg raid6 volume(s) ********* RAID hash info for this scenario ********* * names: synced_primary_raid6_4legs_1 * sync: 1 * type: raid6 * -m |-i value: 4 * leg devices: /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdd1 /dev/sdg1 /dev/sdb1 * spanned legs: 0 * manual repair: 0 * failpv(s): /dev/sde1 * failnode(s): host-123 * lvmetad: 1 * cache stack: 1 * raid fault policy: allocate ****************************************************** Creating raids(s) on host-123... host-123: lvcreate --type raid6 -i 4 -n synced_primary_raid6_4legs_1 -L 500M black_bird /dev/sde1:0-2400 /dev/sda1:0-2400 /dev/sdf1:0-2400 /dev/sdd1:0-2400 /dev/sdg1:0-2400 /dev/sdb1:0-2400 Current mirror/raid device structure(s): LV Attr LSize Cpy%Sync Devices synced_primary_raid6_4legs_1 rwi-aor--- 512.00m 18.75 synced_primary_raid6_4legs_1_rimage_0(0),synced_primary_raid6_4legs_1_rimage_1(0),synced_primary_raid6_4legs_1_rimage_2(0),synced_primary_raid6_4legs_1_rimage_3(0),synced_primary_raid6_4legs_1_rimage_4(0),synced_primary_raid6_4legs_1_rimage_5(0) [synced_primary_raid6_4legs_1_rimage_0] Iwi-aor--- 128.00m /dev/sde1(1) [synced_primary_raid6_4legs_1_rimage_1] Iwi-aor--- 128.00m /dev/sda1(1) [synced_primary_raid6_4legs_1_rimage_2] Iwi-aor--- 128.00m /dev/sdf1(1) [synced_primary_raid6_4legs_1_rimage_3] Iwi-aor--- 128.00m /dev/sdd1(1) [synced_primary_raid6_4legs_1_rimage_4] Iwi-aor--- 128.00m /dev/sdg1(1) [synced_primary_raid6_4legs_1_rimage_5] Iwi-aor--- 128.00m /dev/sdb1(1) [synced_primary_raid6_4legs_1_rmeta_0] ewi-aor--- 4.00m /dev/sde1(0) [synced_primary_raid6_4legs_1_rmeta_1] ewi-aor--- 4.00m /dev/sda1(0) [synced_primary_raid6_4legs_1_rmeta_2] ewi-aor--- 4.00m /dev/sdf1(0) [synced_primary_raid6_4legs_1_rmeta_3] ewi-aor--- 4.00m /dev/sdd1(0) [synced_primary_raid6_4legs_1_rmeta_4] ewi-aor--- 4.00m /dev/sdg1(0) [synced_primary_raid6_4legs_1_rmeta_5] ewi-aor--- 4.00m /dev/sdb1(0) Waiting until all mirror|raid volumes become fully syncd... 1/1 mirror(s) are fully synced: ( 100.00% ) Sleeping 15 sec ********* CACHE info for this scenario ********* * Killing the raid cache ORIGIN (slow) device * cache pool device: /dev/sda1 * cachemode: writethrough **************************************************** Convert mirror/raid volume(s) to Cache volume(s) on host-123... Creating CACHE DATA and META (fast) devices (which will not be failed), convert to cache pool create a cache volume lvcreate -n cache_synced_primary_raid6_4legs_1 -L 500M black_bird /dev/sda1 WARNING: xfs signature detected on /dev/black_bird/cache_synced_primary_raid6_4legs_1 at offset 0. Wipe it? [y/n]: [n] Aborted wiping of xfs. 1 existing signature left on the device. lvcreate -n meta -L 12M black_bird /dev/sda1 Create cache pool volume by combining the cache data and cache metadata (fast) volumes lvconvert --yes --type cache-pool --cachemode writethrough --poolmetadata black_bird/meta black_bird/cache_synced_primary_raid6_4legs_1 WARNING: Converting logical volume black_bird/cache_synced_primary_raid6_4legs_1 and black_bird/meta to cache pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Create cached volume by combining the cache pool (fast) and origin (slow) volumes lvconvert --yes --type cache --cachepool black_bird/cache_synced_primary_raid6_4legs_1 black_bird/synced_primary_raid6_4legs_1 Creating xfs on top of mirror(s) on host-123... Mounting mirrored xfs filesystems on host-123... Current mirror/raid device structure(s): LV Attr LSize Cpy%Sync Devices [cache_synced_primary_raid6_4legs_1] Cwi---C--- 500.00m 0.00 cache_synced_primary_raid6_4legs_1_cdata(0) [cache_synced_primary_raid6_4legs_1_cdata] Cwi-ao---- 500.00m /dev/sda1(33) [cache_synced_primary_raid6_4legs_1_cmeta] ewi-ao---- 12.00m /dev/sda1(158) [lvol0_pmspare] ewi------- 12.00m /dev/sde1(33) synced_primary_raid6_4legs_1 Cwi-aoC--- 512.00m 0.00 synced_primary_raid6_4legs_1_corig(0) [synced_primary_raid6_4legs_1_corig] rwi-aoC--- 512.00m 100.00 synced_primary_raid6_4legs_1_corig_rimage_0(0),synced_primary_raid6_4legs_1_corig_rimage_1(0),synced_primary_raid6_4legs_1_corig_rimage_2(0),synced_primary_raid6_4legs_1_corig_rimage_3(0),synced_primary_raid6_4legs_1_corig_rimage_4(0),synced_primary_raid6_4legs_1_corig_rimage_5(0) [synced_primary_raid6_4legs_1_corig_rimage_0] iwi-aor--- 128.00m /dev/sde1(1) [synced_primary_raid6_4legs_1_corig_rimage_1] iwi-aor--- 128.00m /dev/sda1(1) [synced_primary_raid6_4legs_1_corig_rimage_2] iwi-aor--- 128.00m /dev/sdf1(1) [synced_primary_raid6_4legs_1_corig_rimage_3] iwi-aor--- 128.00m /dev/sdd1(1) [synced_primary_raid6_4legs_1_corig_rimage_4] iwi-aor--- 128.00m /dev/sdg1(1) [synced_primary_raid6_4legs_1_corig_rimage_5] iwi-aor--- 128.00m /dev/sdb1(1) [synced_primary_raid6_4legs_1_corig_rmeta_0] ewi-aor--- 4.00m /dev/sde1(0) [synced_primary_raid6_4legs_1_corig_rmeta_1] ewi-aor--- 4.00m /dev/sda1(0) [synced_primary_raid6_4legs_1_corig_rmeta_2] ewi-aor--- 4.00m /dev/sdf1(0) [synced_primary_raid6_4legs_1_corig_rmeta_3] ewi-aor--- 4.00m /dev/sdd1(0) [synced_primary_raid6_4legs_1_corig_rmeta_4] ewi-aor--- 4.00m /dev/sdg1(0) [synced_primary_raid6_4legs_1_corig_rmeta_5] ewi-aor--- 4.00m /dev/sdb1(0) PV=/dev/sde1 lvol0_pmspare: 1.0 synced_primary_raid6_4legs_1_corig_rimage_0: 1.0 synced_primary_raid6_4legs_1_corig_rmeta_0: 1.0 Writing verification files (checkit) to mirror(s) on... ---- host-123 ---- <start name="host-123_synced_primary_raid6_4legs_1" pid="7258" time="Thu Sep 29 14:51:18 2016" type="cmd" /> Sleeping 15 seconds to get some outsanding I/O locks before the failure Verifying files (checkit) on mirror(s) on... ---- host-123 ---- Disabling device sde on host-123rescan device... /dev/sde1: read failed after 0 of 512 at 21467824128: Input/output error /dev/sde1: read failed after 0 of 512 at 21467938816: Input/output error /dev/sde1: read failed after 0 of 512 at 0: Input/output error /dev/sde1: read failed after 0 of 512 at 4096: Input/output error /dev/sde1: read failed after 0 of 2048 at 0: Input/output error Getting recovery check start time from /var/log/messages: Sep 29 14:51 Attempting I/O to cause mirror down conversion(s) on host-123 dd if=/dev/zero of=/mnt/synced_primary_raid6_4legs_1/ddfile count=10 bs=4M 10+0 records in 10+0 records out 41943040 bytes (42 MB) copied, 0.173477 s, 242 MB/s Verifying current sanity of lvm after the failure Current mirror/raid device structure(s): WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. LV Attr LSize Cpy%Sync Devices [cache_synced_primary_raid6_4legs_1] Cwi---C--- 500.00m 0.00 cache_synced_primary_raid6_4legs_1_cdata(0) [cache_synced_primary_raid6_4legs_1_cdata] Cwi-ao---- 500.00m /dev/sda1(33) [cache_synced_primary_raid6_4legs_1_cmeta] ewi-ao---- 12.00m /dev/sda1(158) [lvol0_pmspare] ewi-----p- 12.00m [unknown](33) synced_primary_raid6_4legs_1 Cwi-aoC-p- 512.00m 0.00 synced_primary_raid6_4legs_1_corig(0) [synced_primary_raid6_4legs_1_corig] rwi-aoC-p- 512.00m 100.00 synced_primary_raid6_4legs_1_corig_rimage_0(0),synced_primary_raid6_4legs_1_corig_rimage_1(0),synced_primary_raid6_4legs_1_corig_rimage_2(0),synced_primary_raid6_4legs_1_corig_rimage_3(0),synced_primary_raid6_4legs_1_corig_rimage_4(0),synced_primary_raid6_4legs_1_corig_rimage_5(0) [synced_primary_raid6_4legs_1_corig_rimage_0] iwi-aor-p- 128.00m [unknown](1) [synced_primary_raid6_4legs_1_corig_rimage_1] iwi-aor--- 128.00m /dev/sda1(1) [synced_primary_raid6_4legs_1_corig_rimage_2] iwi-aor--- 128.00m /dev/sdf1(1) [synced_primary_raid6_4legs_1_corig_rimage_3] iwi-aor--- 128.00m /dev/sdd1(1) [synced_primary_raid6_4legs_1_corig_rimage_4] iwi-aor--- 128.00m /dev/sdg1(1) [synced_primary_raid6_4legs_1_corig_rimage_5] iwi-aor--- 128.00m /dev/sdb1(1) [synced_primary_raid6_4legs_1_corig_rmeta_0] ewi-aor-p- 4.00m [unknown](0) [synced_primary_raid6_4legs_1_corig_rmeta_1] ewi-aor--- 4.00m /dev/sda1(0) [synced_primary_raid6_4legs_1_corig_rmeta_2] ewi-aor--- 4.00m /dev/sdf1(0) [synced_primary_raid6_4legs_1_corig_rmeta_3] ewi-aor--- 4.00m /dev/sdd1(0) [synced_primary_raid6_4legs_1_corig_rmeta_4] ewi-aor--- 4.00m /dev/sdg1(0) [synced_primary_raid6_4legs_1_corig_rmeta_5] ewi-aor--- 4.00m /dev/sdb1(0) Verifying FAILED device /dev/sde1 is *NOT* in the volume(s) WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Verifying IMAGE device /dev/sda1 *IS* in the volume(s) WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Verifying IMAGE device /dev/sdf1 *IS* in the volume(s) WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Verifying IMAGE device /dev/sdd1 *IS* in the volume(s) WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Verifying IMAGE device /dev/sdg1 *IS* in the volume(s) WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Verifying IMAGE device /dev/sdb1 *IS* in the volume(s) WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Verify the rimage/rmeta dm devices remain after the failures Checking EXISTENCE and STATE of lvol0_pmspare on: Skipping check for lvol0_pmspare until BUG 1030121 (lvol0_pmspare) is sorted out Checking EXISTENCE and STATE of synced_primary_raid6_4legs_1_corig_rimage_0 on: host-123 (ALLOCATE POLICY) there should not be an 'unknown' device associated with synced_primary_raid6_4legs_1_corig_rimage_0 on host-123 Sep 29 14:51:39 host-123 qarshd[18024]: Running cmdline: echo offline > /sys/block/sde/device/state Sep 29 14:51:39 host-123 systemd: Started qarsh Per-Connection Server (10.15.80.224:50172). Sep 29 14:51:39 host-123 systemd: Starting qarsh Per-Connection Server (10.15.80.224:50172)... Sep 29 14:51:39 host-123 qarshd[18028]: Talking to peer ::ffff:10.15.80.224:50172 (IPv6) Sep 29 14:51:39 host-123 qarshd[18028]: Running cmdline: pvscan --cache /dev/sde1 Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device Sep 29 14:51:39 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device Sep 29 14:51:43 host-123 kernel: sd 4:0:0:1: rejecting I/O to offline device Sep 29 14:51:43 host-123 kernel: md: super_written gets error=-5, uptodate=0 Sep 29 14:51:43 host-123 kernel: md/raid:mdX: Disk failure on dm-3, disabling device.#012md/raid:mdX: Operation continuing on 5 devices. Sep 29 14:51:43 host-123 lvm[16364]: Device #0 of raid6_zr array, black_bird-synced_primary_raid6_4legs_1_corig, has failed. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr already missing, skipping. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Device for PV eX9fOz-z6UL-sJXu-ErHI-i7nS-HaPV-LkhCgr not found or rejected by a filter. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rimage_0 while checking used and assumed devices. Sep 29 14:51:43 host-123 lvm[16364]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid6_4legs_1_corig_rmeta_0 while checking used and assumed devices. Sep 29 14:51:43 host-123 lvm[16364]: Cannot convert internal LV black_bird/synced_primary_raid6_4legs_1_corig. Sep 29 14:51:43 host-123 lvm[16364]: Failed to process event for black_bird-synced_primary_raid6_4legs_1_corig. Version-Release number of selected component (if applicable): 3.10.0-510.el7.x86_64 lvm2-2.02.166-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 lvm2-libs-2.02.166-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 lvm2-cluster-2.02.166-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 device-mapper-1.02.135-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 device-mapper-libs-1.02.135-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 device-mapper-event-1.02.135-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 device-mapper-event-libs-1.02.135-1.el7 BUILT: Wed Sep 28 02:26:52 CDT 2016 device-mapper-persistent-data-0.6.3-1.el7 BUILT: Fri Jul 22 05:29:13 CDT 2016
This appear to only affect cache raid repair when the fault policy is "allocate", warn with a manual repair attempt appears to succeed.
Reproduced with raid_fault_policy="allocate". "lvconvert --repair black_bird/synced_primary_raid6_4legs_1_corig" fails here with "Cannot convert internal LV ...". Is this a regression at all?
1. verified that a manual repair is never attempted due to the restriction that cache and pool raids need to be inactive during the passing 'warn' cases, so disregard comment #1. 2. verified that this same test case passed in 7.2.z as well as in a previous version of rhel7.3: 3.10.0-510.el7.x86_64 lvm2-2.02.161-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 lvm2-libs-2.02.161-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 lvm2-cluster-2.02.161-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 device-mapper-1.02.131-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 device-mapper-libs-1.02.131-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 device-mapper-event-1.02.131-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 device-mapper-event-libs-1.02.131-3.el7 BUILT: Thu Jul 28 09:31:24 CDT 2016 device-mapper-persistent-data-0.6.3-1.el7 BUILT: Fri Jul 22 05:29:13 CDT 2016 Oct 4 11:17:01 host-116 lvm[3815]: Device #0 of raid1 array, black_bird-synced_primary_raid1_2legs_1_cdata, has failed. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rimage_0 while checking used and assumed devices. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rmeta_0 while checking used and assumed devices. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V already missing, skipping. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rimage_0 while checking used and assumed devices. Oct 4 11:17:01 host-116 lvm[3815]: WARNING: Couldn't find all devices for LV black_bird/synced_primary_raid1_2legs_1_cdata_rmeta_0 while checking used and assumed devices. Oct 4 11:17:02 host-116 kernel: device-mapper: raid: Device 0 specified for rebuild; clearing superblock Oct 4 11:17:02 host-116 kernel: md/raid1:mdX: active with 1 out of 2 mirrors Oct 4 11:17:02 host-116 kernel: created bitmap (1 pages) for device mdX Oct 4 11:17:03 host-116 kernel: mdX: bitmap initialized from disk: read 1 pages, set 0 of 1000 bits Oct 4 11:17:03 host-116 kernel: md: recovery of RAID array mdX Oct 4 11:17:03 host-116 kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Oct 4 11:17:03 host-116 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery. Oct 4 11:17:03 host-116 kernel: md: using 128k window, over a total of 512000k. Oct 4 11:17:03 host-116 multipathd: dm-2: remove map (uevent) Oct 4 11:17:03 host-116 multipathd: dm-2: devmap not registered, can't remove Oct 4 11:17:03 host-116 multipathd: dm-3: remove map (uevent) Oct 4 11:17:03 host-116 multipathd: dm-3: devmap not registered, can't remove Oct 4 11:17:03 host-116 multipathd: dm-13: remove map (uevent) Oct 4 11:17:03 host-116 multipathd: dm-13: devmap not registered, can't remove Oct 4 11:17:03 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51342). Oct 4 11:17:03 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51342)... Oct 4 11:17:03 host-116 multipathd: dm-12: remove map (uevent) Oct 4 11:17:03 host-116 multipathd: dm-12: devmap not registered, can't remove Oct 4 11:17:03 host-116 multipathd: dm-2: remove map (uevent) Oct 4 11:17:03 host-116 multipathd: dm-3: remove map (uevent) Oct 4 11:17:03 host-116 qarshd[6698]: Talking to peer ::ffff:10.15.80.224:51342 (IPv6) Oct 4 11:17:03 host-116 multipathd: dm-12: remove map (uevent) Oct 4 11:17:03 host-116 multipathd: dm-13: remove map (uevent) Oct 4 11:17:03 host-116 qarshd[6698]: Running cmdline: pvs -a Oct 4 11:17:03 host-116 kernel: md/raid1:mdX: active with 1 out of 2 mirrors Oct 4 11:17:03 host-116 kernel: created bitmap (1 pages) for device mdX Oct 4 11:17:04 host-116 kernel: md: mdX: recovery interrupted. Oct 4 11:17:04 host-116 kernel: mdX: bitmap initialized from disk: read 1 pages, set 0 of 1000 bits Oct 4 11:17:04 host-116 kernel: md: recovery of RAID array mdX Oct 4 11:17:04 host-116 kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Oct 4 11:17:04 host-116 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery. Oct 4 11:17:04 host-116 kernel: md: using 128k window, over a total of 512000k. Oct 4 11:17:04 host-116 kernel: md: resuming recovery of mdX from checkpoint. Oct 4 11:17:04 host-116 lvm[3815]: Faulty devices in black_bird/synced_primary_raid1_2legs_1_cdata successfully replaced. Oct 4 11:17:04 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51344). Oct 4 11:17:04 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51344)... Oct 4 11:17:04 host-116 qarshd[6751]: Talking to peer ::ffff:10.15.80.224:51344 (IPv6) Oct 4 11:17:04 host-116 qarshd[6751]: Running cmdline: tail -n1 /var/log/messages | cut -c 1-12 Oct 4 11:17:04 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51346). Oct 4 11:17:04 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51346)... Oct 4 11:17:04 host-116 qarshd[6757]: Talking to peer ::ffff:10.15.80.224:51346 (IPv6) Oct 4 11:17:05 host-116 qarshd[6757]: Running cmdline: dd if=/dev/zero of=/mnt/synced_primary_raid1_2legs_1/ddfile count=10 bs=4M Oct 4 11:17:05 host-116 systemd: Started qarsh Per-Connection Server (10.15.80.224:51348). Oct 4 11:17:05 host-116 systemd: Starting qarsh Per-Connection Server (10.15.80.224:51348)... Oct 4 11:17:05 host-116 qarshd[6761]: Talking to peer ::ffff:10.15.80.224:51348 (IPv6) Oct 4 11:17:05 host-116 qarshd[6761]: Running cmdline: sync Oct 4 11:17:08 host-116 kernel: md: mdX: recovery done. Oct 4 11:17:08 host-116 lvm[3815]: Device #0 of raid1 array, black_bird-synced_primary_raid1_2legs_1_cdata, has failed. Oct 4 11:17:08 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter. Oct 4 11:17:08 host-116 lvm[3815]: WARNING: Device for PV dlewYC-gbDe-njA2-GeRl-jnev-zJ0I-p92C5V not found or rejected by a filter. Oct 4 11:17:08 host-116 lvm[3815]: Faulty devices in black_bird/synced_primary_raid1_2legs_1_cdata successfully replaced.
There is a work-around for this issue (I'll elaborate on that in next comment). It should not cause hangs/corruptions - only inconvenience. As such, I am pushing for 7.4 fix with 7.3.z inclusion.
Got one-liner fix to lvconvert.c allowing for the rebuild of corig internal LV in testing...
Fix pushed upstream: allow repair on cache orig RAID LVs and "lvconvert --replace/--mirrors/--type {raid*|mirror|striped|linear}" as well. Allow the same lvconvert actions on any cache pool and metadata RAID SubLVs.
Created attachment 1272293 [details] test result Thank you for explanation, I was not aware of this behaviour. Marking verified. 3.10.0-640.el7.x86_64 lvm2-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 lvm2-libs-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 lvm2-cluster-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-libs-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-event-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-event-libs-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 17:15:46 CEST 2017 cmirror-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2222