Bug 1420550

Summary: lvmetad PV confusion led to activation failure of transiently failed raid volume
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Heinz Mauelshagen <heinzm>
lvm2 sub component: Mirroring and RAID (RHEL6) QA Contact: cluster-qe <cluster-qe>
Status: CLOSED WONTFIX Docs Contact:
Severity: low    
Priority: unspecified CC: agk, heinzm, jbrassow, msnitzer, prajnoha, prockai, zkabelac
Version: 6.9   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-06 11:57:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2017-02-08 23:48:26 UTC
Description of problem:
================================================================================                                                                                                                                                                                      
Iteration 0.16 started at Wed Feb  8 16:45:00 CST 2017
================================================================================
Scenario kill_second_spanned_primary_synced_raid4_2legs: Kill primary leg of synced 2 leg raid4 volume(s)

********* RAID hash info for this scenario *********
* names:              synced_spanned_primary_raid4_2legs_1
* sync:               1
* type:               raid4
* -m |-i value:       2
* leg devices:        /dev/sda1 /dev/sdd1 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdg1
* spanned legs:       1
* manual repair:      0
* failpv(s):          /dev/sdc1
* failnode(s):        host-082
* lvmetad:            1
* raid fault policy:  warn
******************************************************

Creating raids(s) on host-082...
host-082: lvcreate --type raid4 -i 2 -n synced_spanned_primary_raid4_2legs_1 -L 500M black_bird /dev/sda1:0-62 /dev/sdd1:0-62 /dev/sdf1:0-62 /dev/sdc1:0-62 /dev/sdb1:0-62 /dev/sdg1:0-62

Current mirror/raid device structure(s):
  LV                                              Attr       LSize   Cpy%Sync Devices
  synced_spanned_primary_raid4_2legs_1            rwi-a-r--- 504.00m 0.00     synced_spanned_primary_raid4_2legs_1_rimage_0(0),synced_spanned_primary_raid4_2legs_1_rimage_1(0),synced_spanned_primary_raid4_2legs_1_rimage_2(0)
  [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi-aor--- 252.00m          /dev/sda1(1)
  [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi-aor--- 252.00m          /dev/sdc1(0)
  [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi-aor--- 252.00m          /dev/sdd1(1)
  [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi-aor--- 252.00m          /dev/sdb1(0)
  [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi-aor--- 252.00m          /dev/sdf1(1)
  [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi-aor--- 252.00m          /dev/sdg1(0)
  [synced_spanned_primary_raid4_2legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sda1(0)
  [synced_spanned_primary_raid4_2legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sdd1(0)
  [synced_spanned_primary_raid4_2legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdf1(0)


Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )

Creating gfs2 on top of mirror(s) on host-082...
mkfs.gfs2 -J 32M -j 1 -p lock_nolock /dev/black_bird/synced_spanned_primary_raid4_2legs_1 -O
Mounting mirrored gfs2 filesystems on host-082...

PV=/dev/sdc1
        synced_spanned_primary_raid4_2legs_1_rimage_0: 2

Writing verification files (checkit) to mirror(s) on...
        ---- host-082 ----


<start name="host-082_synced_spanned_primary_raid4_2legs_1"  pid="13692" time="Wed Feb  8 16:45:40 2017 -0600" type="cmd" />
Sleeping 15 seconds to get some outsanding I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- host-082 ----


Disabling device sdc on host-082

Attempting I/O to cause mirror down conversion(s) on host-082
dd if=/dev/zero of=/mnt/synced_spanned_primary_raid4_2legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.207122 s, 203 MB/s
dd if=/dev/zero of=/mnt/synced_spanned_primary_raid4_2legs_1/ddfile seek=200 count=50 bs=1M
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 2.1289 s, 24.6 MB/s

HACK TO KILL XDOIO...
<fail name="host-082_synced_spanned_primary_raid4_2legs_1"  pid="13692" time="Wed Feb  8 16:46:11 2017 -0600" type="cmd" duration="31" ec="143" />
ALL STOP!
Unmounting gfs and removing mnt point on host-082...

Verifying proper "D"ead kernel status state for failed raid images(s)
No dead "D" kernel state was found for this raid image
This is a known issue where triggering a repair in raid4|5 spanned volumes is difficult and inconsistent, moving on...

Reactivaing the raids containing transiently failed raid images
lvchange -an black_bird/synced_spanned_primary_raid4_2legs_1

lvchange -ay  black_bird/synced_spanned_primary_raid4_2legs_1
  /dev/sdc1: open failed: No such device or address
  Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
  One or more devices used as PVs in VG black_bird have changed sizes.
  device-mapper: reload ioctl on (253:3) failed: No such device or address
unable to re-activate black_bird/synced_spanned_primary_raid4_2legs_1




[root@host-082 ~]# lvs -a -o +devices
  /dev/sdc1: open failed: No such device or address
  Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
  One or more devices used as PVs in VG black_bird have changed sizes.
  LV                                              Attr       LSize   Cpy%Sync Devices
  synced_spanned_primary_raid4_2legs_1            rwi---r--- 504.00m          synced_spanned_primary_raid4_2legs_1_rimage_0(0),synced_spanned_primary_raid4_2legs_1_rimage_1(0),synced_spanned_primary_raid4_2legs_1_rimage_2(0)
  [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi---r--- 252.00m          /dev/sda1(1)
  [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi---r--- 252.00m          /dev/sdc1(0)
  [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi---r--- 252.00m          /dev/sdd1(1)
  [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi---r--- 252.00m          /dev/sdb1(0)
  [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi---r--- 252.00m          /dev/sdf1(1)
  [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi---r--- 252.00m          /dev/sdg1(0)
  [synced_spanned_primary_raid4_2legs_1_rmeta_0]  ewi-a-r-r-   4.00m          /dev/sda1(0)
  [synced_spanned_primary_raid4_2legs_1_rmeta_1]  ewi---r---   4.00m          /dev/sdd1(0)
  [synced_spanned_primary_raid4_2legs_1_rmeta_2]  ewi---r---   4.00m          /dev/sdf1(0)

[root@host-082 ~]# lvchange -ay  black_bird/synced_spanned_primary_raid4_2legs_1
  /dev/sdc1: open failed: No such device or address
  Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
  One or more devices used as PVs in VG black_bird have changed sizes.
  device-mapper: reload ioctl on (253:3) failed: No such device or address

[root@host-082 ~]# dmsetup ls
black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0 (253:2)
[root@host-082 ~]# dmsetup remove black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0

[root@host-082 ~]# lvchange -ay  black_bird/synced_spanned_primary_raid4_2legs_1
  /dev/sdc1: open failed: No such device or address
  Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
  One or more devices used as PVs in VG black_bird have changed sizes.
  device-mapper: reload ioctl on (253:3) failed: No such device or address

[root@host-082 ~]# dmsetup remove black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0
[root@host-082 ~]# pvscan --cache

# Now it works after the pvscan
[root@host-082 ~]# lvchange -ay  black_bird/synced_spanned_primary_raid4_2legs_1
  WARNING: Device for PV G3DKdR-Jt8T-HeuK-L3HW-TCCy-SNg5-Qnibiu not found or rejected by a filter.


Version-Release number of selected component (if applicable):
2.6.32-688.el6.x86_64

lvm2-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
lvm2-libs-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
lvm2-cluster-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
udev-147-2.73.el6_8.2    BUILT: Tue Aug 30 08:17:19 CDT 2016
device-mapper-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-libs-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-event-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-event-libs-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-persistent-data-0.6.2-0.1.rc7.el6    BUILT: Tue Mar 22 08:58:09 CDT 2016


How reproducible:
Only once so far

Comment 2 Jan Kurik 2017-12-06 11:57:44 UTC
Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available.

The official life cycle policy can be reviewed here:

http://redhat.com/rhel/lifecycle

This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL:

https://access.redhat.com/