801613 – Failed to replace faulty raid devices: Reshaping arrays not yet supported

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 801613 - Failed to replace faulty raid devices: Reshaping arrays not yet supported

Summary: Failed to replace faulty raid devices: Reshaping arrays not yet supported

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	6.3
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Jonathan Earl Brassow
QA Contact:	Cluster QE
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-03-09 00:23 UTC by Corey Marthaler
Modified:	2012-03-15 14:55 UTC (History)
CC List:	10 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-03-15 14:55:47 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Corey Marthaler 2012-03-09 00:23:37 UTC

Description of problem:
I'm seeing this issue while running raid image failure testing with allocation turned on. This may be related to bug 801571.

Scenario kill_random_synced_raid5_3legs: Kill random leg of synced 3 leg raid5 volume(s)

********* RAID hash info for this scenario *********
* names:              synced_random_raid5_3legs_1
* sync:               1
* type:               raid5
* -m |-i value:       3
* leg devices:        /dev/sdg1 /dev/sdf1 /dev/sde1 /dev/sdc1
* failpv(s):          /dev/sdc1
* failnode(s):        taft-01
* raid fault policy:   allocate
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid5 -i 3 -n synced_random_raid5_3legs_1 -L 500M black_bird /dev/sdg1:0-1000 /dev/sdf1:0-1000 /dev/sde1:0-1000 /dev/sdc1:0-1000

RAID Structure(s):
  LV                                     Attr     LSize   Copy%  Devices
  synced_random_raid5_3legs_1            rwi-a-r- 504.00m        synced_random_raid5_3legs_1_rimage_0(0),synced_random_raid5_3legs_1_rimage_1(0),synced_random_raid5_3legs_1_rimage_2(0),synced_random_raid5_3legs_1_rimage_3(0)
  [synced_random_raid5_3legs_1_rimage_0] Iwi-aor- 168.00m        /dev/sdg1(1)
  [synced_random_raid5_3legs_1_rimage_1] Iwi-aor- 168.00m        /dev/sdf1(1)
  [synced_random_raid5_3legs_1_rimage_2] Iwi-aor- 168.00m        /dev/sde1(1)
  [synced_random_raid5_3legs_1_rimage_3] Iwi-aor- 168.00m        /dev/sdc1(1)
  [synced_random_raid5_3legs_1_rmeta_0]  ewi-aor-   4.00m        /dev/sdg1(0)
  [synced_random_raid5_3legs_1_rmeta_1]  ewi-aor-   4.00m        /dev/sdf1(0)
  [synced_random_raid5_3legs_1_rmeta_2]  ewi-aor-   4.00m        /dev/sde1(0)
  [synced_random_raid5_3legs_1_rmeta_3]  ewi-aor-   4.00m        /dev/sdc1(0)

PVS IN VG: /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
PV=/dev/sdc1
     synced_random_raid5_3legs_1_rimage_3: 1
     synced_random_raid5_3legs_1_rmeta_3: 1

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
     ---- taft-01 ----

Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
     ---- taft-01 ----

Disabling device sdc on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.536352 s, 78.2 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
  /dev/sdc1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdc1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid LOdwxl-cgjB-XCHH-e4j0-FNry-aZlG-D3Edvk.
  LV                                     Attr     LSize   Copy%  Devices
  synced_random_raid5_3legs_1            rwi-aor- 504.00m        synced_random_raid5_3legs_1_rimage_0(0),synced_random_raid5_3legs_1_rimage_1(0),synced_random_raid5_3legs_1_rimage_2(0),synced_random_raid5_3legs_1_rimage_3(0)
  [synced_random_raid5_3legs_1_rimage_0] iwi-aor- 168.00m        /dev/sdg1(1)
  [synced_random_raid5_3legs_1_rimage_1] iwi-aor- 168.00m        /dev/sdf1(1)
  [synced_random_raid5_3legs_1_rimage_2] iwi-aor- 168.00m        /dev/sde1(1)
  [synced_random_raid5_3legs_1_rimage_3] iwi-aor- 168.00m        unknown device(1)
  [synced_random_raid5_3legs_1_rmeta_0]  ewi-aor-   4.00m        /dev/sdg1(0)
  [synced_random_raid5_3legs_1_rmeta_1]  ewi-aor-   4.00m        /dev/sdf1(0)
  [synced_random_raid5_3legs_1_rmeta_2]  ewi-aor-   4.00m        /dev/sde1(0)
  [synced_random_raid5_3legs_1_rmeta_3]  ewi-aor-   4.00m        unknown device(0)

Verifying FAILED device /dev/sdc1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdg1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdf1 *IS* in the volume(s)
Verifying IMAGE device /dev/sde1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid5_3legs_1_rimage_3 on:  taft-01there should not be an 'unknown' device associated with synced_random_raid5_3legs_1_rimage_3 on taft-01



Mar  8 17:51:20 taft-01 kernel: device-mapper: raid: Reshaping arrays not yet supported.
Mar  8 17:51:20 taft-01 kernel: device-mapper: table: 253:11: raid: Unable to assemble array: Invalid superblocks
Mar  8 17:51:20 taft-01 kernel: device-mapper: ioctl: error adding target to table
Mar  8 17:51:20 taft-01 lvm[3262]: device-mapper: reload ioctl on  failed: Invalid argument
Mar  8 17:51:20 taft-01 lvm[3262]: Failed to suspend black_bird/synced_random_raid5_3legs_1 before committing changes
Mar  8 17:51:20 taft-01 lvm[3262]: Failed to replace faulty devices in black_bird/synced_random_raid5_3legs_1.
Mar  8 17:51:20 taft-01 lvm[3262]: Repair of RAID device black_bird-synced_random_raid5_3legs_1 failed.
Mar  8 17:51:20 taft-01 lvm[3262]: Failed to process event for black_bird-synced_random_raid5_3legs_1


Version-Release number of selected component (if applicable):
2.6.32-220.4.2.el6.x86_64

lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012


How reproducible:
Often

Comment 1 Corey Marthaler 2012-03-15 14:55:47 UTC

I have not been able to reproduce this since updating the kernel. Closing and
will reopen if seen again in the future...

2.6.32-251.el6.x86_64
lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012

Note You need to log in before you can comment on or make changes to this bug.