Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionCorey Marthaler
2017-02-08 23:48:26 UTC
Description of problem:
================================================================================
Iteration 0.16 started at Wed Feb 8 16:45:00 CST 2017
================================================================================
Scenario kill_second_spanned_primary_synced_raid4_2legs: Kill primary leg of synced 2 leg raid4 volume(s)
********* RAID hash info for this scenario *********
* names: synced_spanned_primary_raid4_2legs_1
* sync: 1
* type: raid4
* -m |-i value: 2
* leg devices: /dev/sda1 /dev/sdd1 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdg1
* spanned legs: 1
* manual repair: 0
* failpv(s): /dev/sdc1
* failnode(s): host-082
* lvmetad: 1
* raid fault policy: warn
******************************************************
Creating raids(s) on host-082...
host-082: lvcreate --type raid4 -i 2 -n synced_spanned_primary_raid4_2legs_1 -L 500M black_bird /dev/sda1:0-62 /dev/sdd1:0-62 /dev/sdf1:0-62 /dev/sdc1:0-62 /dev/sdb1:0-62 /dev/sdg1:0-62
Current mirror/raid device structure(s):
LV Attr LSize Cpy%Sync Devices
synced_spanned_primary_raid4_2legs_1 rwi-a-r--- 504.00m 0.00 synced_spanned_primary_raid4_2legs_1_rimage_0(0),synced_spanned_primary_raid4_2legs_1_rimage_1(0),synced_spanned_primary_raid4_2legs_1_rimage_2(0)
[synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi-aor--- 252.00m /dev/sda1(1)
[synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi-aor--- 252.00m /dev/sdc1(0)
[synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi-aor--- 252.00m /dev/sdd1(1)
[synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi-aor--- 252.00m /dev/sdb1(0)
[synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi-aor--- 252.00m /dev/sdf1(1)
[synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi-aor--- 252.00m /dev/sdg1(0)
[synced_spanned_primary_raid4_2legs_1_rmeta_0] ewi-aor--- 4.00m /dev/sda1(0)
[synced_spanned_primary_raid4_2legs_1_rmeta_1] ewi-aor--- 4.00m /dev/sdd1(0)
[synced_spanned_primary_raid4_2legs_1_rmeta_2] ewi-aor--- 4.00m /dev/sdf1(0)
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Creating gfs2 on top of mirror(s) on host-082...
mkfs.gfs2 -J 32M -j 1 -p lock_nolock /dev/black_bird/synced_spanned_primary_raid4_2legs_1 -O
Mounting mirrored gfs2 filesystems on host-082...
PV=/dev/sdc1
synced_spanned_primary_raid4_2legs_1_rimage_0: 2
Writing verification files (checkit) to mirror(s) on...
---- host-082 ----
<start name="host-082_synced_spanned_primary_raid4_2legs_1" pid="13692" time="Wed Feb 8 16:45:40 2017 -0600" type="cmd" />
Sleeping 15 seconds to get some outsanding I/O locks before the failure
Verifying files (checkit) on mirror(s) on...
---- host-082 ----
Disabling device sdc on host-082
Attempting I/O to cause mirror down conversion(s) on host-082
dd if=/dev/zero of=/mnt/synced_spanned_primary_raid4_2legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.207122 s, 203 MB/s
dd if=/dev/zero of=/mnt/synced_spanned_primary_raid4_2legs_1/ddfile seek=200 count=50 bs=1M
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 2.1289 s, 24.6 MB/s
HACK TO KILL XDOIO...
<fail name="host-082_synced_spanned_primary_raid4_2legs_1" pid="13692" time="Wed Feb 8 16:46:11 2017 -0600" type="cmd" duration="31" ec="143" />
ALL STOP!
Unmounting gfs and removing mnt point on host-082...
Verifying proper "D"ead kernel status state for failed raid images(s)
No dead "D" kernel state was found for this raid image
This is a known issue where triggering a repair in raid4|5 spanned volumes is difficult and inconsistent, moving on...
Reactivaing the raids containing transiently failed raid images
lvchange -an black_bird/synced_spanned_primary_raid4_2legs_1
lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1
/dev/sdc1: open failed: No such device or address
Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
One or more devices used as PVs in VG black_bird have changed sizes.
device-mapper: reload ioctl on (253:3) failed: No such device or address
unable to re-activate black_bird/synced_spanned_primary_raid4_2legs_1
[root@host-082 ~]# lvs -a -o +devices
/dev/sdc1: open failed: No such device or address
Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
One or more devices used as PVs in VG black_bird have changed sizes.
LV Attr LSize Cpy%Sync Devices
synced_spanned_primary_raid4_2legs_1 rwi---r--- 504.00m synced_spanned_primary_raid4_2legs_1_rimage_0(0),synced_spanned_primary_raid4_2legs_1_rimage_1(0),synced_spanned_primary_raid4_2legs_1_rimage_2(0)
[synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi---r--- 252.00m /dev/sda1(1)
[synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi---r--- 252.00m /dev/sdc1(0)
[synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi---r--- 252.00m /dev/sdd1(1)
[synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi---r--- 252.00m /dev/sdb1(0)
[synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi---r--- 252.00m /dev/sdf1(1)
[synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi---r--- 252.00m /dev/sdg1(0)
[synced_spanned_primary_raid4_2legs_1_rmeta_0] ewi-a-r-r- 4.00m /dev/sda1(0)
[synced_spanned_primary_raid4_2legs_1_rmeta_1] ewi---r--- 4.00m /dev/sdd1(0)
[synced_spanned_primary_raid4_2legs_1_rmeta_2] ewi---r--- 4.00m /dev/sdf1(0)
[root@host-082 ~]# lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1
/dev/sdc1: open failed: No such device or address
Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
One or more devices used as PVs in VG black_bird have changed sizes.
device-mapper: reload ioctl on (253:3) failed: No such device or address
[root@host-082 ~]# dmsetup ls
black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0 (253:2)
[root@host-082 ~]# dmsetup remove black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0
[root@host-082 ~]# lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1
/dev/sdc1: open failed: No such device or address
Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized?
One or more devices used as PVs in VG black_bird have changed sizes.
device-mapper: reload ioctl on (253:3) failed: No such device or address
[root@host-082 ~]# dmsetup remove black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0
[root@host-082 ~]# pvscan --cache
# Now it works after the pvscan
[root@host-082 ~]# lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1
WARNING: Device for PV G3DKdR-Jt8T-HeuK-L3HW-TCCy-SNg5-Qnibiu not found or rejected by a filter.
Version-Release number of selected component (if applicable):
2.6.32-688.el6.x86_64
lvm2-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
lvm2-libs-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
lvm2-cluster-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
udev-147-2.73.el6_8.2 BUILT: Tue Aug 30 08:17:19 CDT 2016
device-mapper-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-libs-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-event-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-event-libs-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-persistent-data-0.6.2-0.1.rc7.el6 BUILT: Tue Mar 22 08:58:09 CDT 2016
How reproducible:
Only once so far
Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available.
The official life cycle policy can be reviewed here:
http://redhat.com/rhel/lifecycle
This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL:
https://access.redhat.com/
Description of problem: ================================================================================ Iteration 0.16 started at Wed Feb 8 16:45:00 CST 2017 ================================================================================ Scenario kill_second_spanned_primary_synced_raid4_2legs: Kill primary leg of synced 2 leg raid4 volume(s) ********* RAID hash info for this scenario ********* * names: synced_spanned_primary_raid4_2legs_1 * sync: 1 * type: raid4 * -m |-i value: 2 * leg devices: /dev/sda1 /dev/sdd1 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdg1 * spanned legs: 1 * manual repair: 0 * failpv(s): /dev/sdc1 * failnode(s): host-082 * lvmetad: 1 * raid fault policy: warn ****************************************************** Creating raids(s) on host-082... host-082: lvcreate --type raid4 -i 2 -n synced_spanned_primary_raid4_2legs_1 -L 500M black_bird /dev/sda1:0-62 /dev/sdd1:0-62 /dev/sdf1:0-62 /dev/sdc1:0-62 /dev/sdb1:0-62 /dev/sdg1:0-62 Current mirror/raid device structure(s): LV Attr LSize Cpy%Sync Devices synced_spanned_primary_raid4_2legs_1 rwi-a-r--- 504.00m 0.00 synced_spanned_primary_raid4_2legs_1_rimage_0(0),synced_spanned_primary_raid4_2legs_1_rimage_1(0),synced_spanned_primary_raid4_2legs_1_rimage_2(0) [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi-aor--- 252.00m /dev/sda1(1) [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi-aor--- 252.00m /dev/sdc1(0) [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi-aor--- 252.00m /dev/sdd1(1) [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi-aor--- 252.00m /dev/sdb1(0) [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi-aor--- 252.00m /dev/sdf1(1) [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi-aor--- 252.00m /dev/sdg1(0) [synced_spanned_primary_raid4_2legs_1_rmeta_0] ewi-aor--- 4.00m /dev/sda1(0) [synced_spanned_primary_raid4_2legs_1_rmeta_1] ewi-aor--- 4.00m /dev/sdd1(0) [synced_spanned_primary_raid4_2legs_1_rmeta_2] ewi-aor--- 4.00m /dev/sdf1(0) Waiting until all mirror|raid volumes become fully syncd... 1/1 mirror(s) are fully synced: ( 100.00% ) Creating gfs2 on top of mirror(s) on host-082... mkfs.gfs2 -J 32M -j 1 -p lock_nolock /dev/black_bird/synced_spanned_primary_raid4_2legs_1 -O Mounting mirrored gfs2 filesystems on host-082... PV=/dev/sdc1 synced_spanned_primary_raid4_2legs_1_rimage_0: 2 Writing verification files (checkit) to mirror(s) on... ---- host-082 ---- <start name="host-082_synced_spanned_primary_raid4_2legs_1" pid="13692" time="Wed Feb 8 16:45:40 2017 -0600" type="cmd" /> Sleeping 15 seconds to get some outsanding I/O locks before the failure Verifying files (checkit) on mirror(s) on... ---- host-082 ---- Disabling device sdc on host-082 Attempting I/O to cause mirror down conversion(s) on host-082 dd if=/dev/zero of=/mnt/synced_spanned_primary_raid4_2legs_1/ddfile count=10 bs=4M 10+0 records in 10+0 records out 41943040 bytes (42 MB) copied, 0.207122 s, 203 MB/s dd if=/dev/zero of=/mnt/synced_spanned_primary_raid4_2legs_1/ddfile seek=200 count=50 bs=1M 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 2.1289 s, 24.6 MB/s HACK TO KILL XDOIO... <fail name="host-082_synced_spanned_primary_raid4_2legs_1" pid="13692" time="Wed Feb 8 16:46:11 2017 -0600" type="cmd" duration="31" ec="143" /> ALL STOP! Unmounting gfs and removing mnt point on host-082... Verifying proper "D"ead kernel status state for failed raid images(s) No dead "D" kernel state was found for this raid image This is a known issue where triggering a repair in raid4|5 spanned volumes is difficult and inconsistent, moving on... Reactivaing the raids containing transiently failed raid images lvchange -an black_bird/synced_spanned_primary_raid4_2legs_1 lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1 /dev/sdc1: open failed: No such device or address Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized? One or more devices used as PVs in VG black_bird have changed sizes. device-mapper: reload ioctl on (253:3) failed: No such device or address unable to re-activate black_bird/synced_spanned_primary_raid4_2legs_1 [root@host-082 ~]# lvs -a -o +devices /dev/sdc1: open failed: No such device or address Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized? One or more devices used as PVs in VG black_bird have changed sizes. LV Attr LSize Cpy%Sync Devices synced_spanned_primary_raid4_2legs_1 rwi---r--- 504.00m synced_spanned_primary_raid4_2legs_1_rimage_0(0),synced_spanned_primary_raid4_2legs_1_rimage_1(0),synced_spanned_primary_raid4_2legs_1_rimage_2(0) [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi---r--- 252.00m /dev/sda1(1) [synced_spanned_primary_raid4_2legs_1_rimage_0] Iwi---r--- 252.00m /dev/sdc1(0) [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi---r--- 252.00m /dev/sdd1(1) [synced_spanned_primary_raid4_2legs_1_rimage_1] Iwi---r--- 252.00m /dev/sdb1(0) [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi---r--- 252.00m /dev/sdf1(1) [synced_spanned_primary_raid4_2legs_1_rimage_2] Iwi---r--- 252.00m /dev/sdg1(0) [synced_spanned_primary_raid4_2legs_1_rmeta_0] ewi-a-r-r- 4.00m /dev/sda1(0) [synced_spanned_primary_raid4_2legs_1_rmeta_1] ewi---r--- 4.00m /dev/sdd1(0) [synced_spanned_primary_raid4_2legs_1_rmeta_2] ewi---r--- 4.00m /dev/sdf1(0) [root@host-082 ~]# lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1 /dev/sdc1: open failed: No such device or address Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized? One or more devices used as PVs in VG black_bird have changed sizes. device-mapper: reload ioctl on (253:3) failed: No such device or address [root@host-082 ~]# dmsetup ls black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0 (253:2) [root@host-082 ~]# dmsetup remove black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0 [root@host-082 ~]# lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1 /dev/sdc1: open failed: No such device or address Device /dev/sdc1 has size of 0 sectors which is smaller than corresponding PV size of 44034102 sectors. Was device resized? One or more devices used as PVs in VG black_bird have changed sizes. device-mapper: reload ioctl on (253:3) failed: No such device or address [root@host-082 ~]# dmsetup remove black_bird-synced_spanned_primary_raid4_2legs_1_rmeta_0 [root@host-082 ~]# pvscan --cache # Now it works after the pvscan [root@host-082 ~]# lvchange -ay black_bird/synced_spanned_primary_raid4_2legs_1 WARNING: Device for PV G3DKdR-Jt8T-HeuK-L3HW-TCCy-SNg5-Qnibiu not found or rejected by a filter. Version-Release number of selected component (if applicable): 2.6.32-688.el6.x86_64 lvm2-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 lvm2-libs-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 lvm2-cluster-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 udev-147-2.73.el6_8.2 BUILT: Tue Aug 30 08:17:19 CDT 2016 device-mapper-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-libs-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-event-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-event-libs-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-persistent-data-0.6.2-0.1.rc7.el6 BUILT: Tue Mar 22 08:58:09 CDT 2016 How reproducible: Only once so far