Bug 825023 - RFE: get rid of the -missing devices during partial allocation failure scenarios
RFE: get rid of the -missing devices during partial allocation failure scenarios
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.3
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 6.4
Assigned To: LVM and device-mapper development team
Corey Marthaler
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-24 16:16 EDT by Corey Marthaler
Modified: 2012-05-24 16:49 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-24 16:49:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2012-05-24 16:16:56 EDT
Description of problem:
./black_bird -o taft-01 -l /home/msp/cmarthal/work/sts/sts-root -r /usr/tests/sts-rhel6.3 -f -i 2

Scenario kill_multiple_synced_raid1_3legs: Kill multiple legs of synced 3 leg raid1 volume(s)

********* RAID hash info for this scenario *********
* names:              synced_multiple_raid1_3legs_1
* sync:               1
* type:               raid1
* -m |-i value:       3
* leg devices:        /dev/sde1 /dev/sdb1 /dev/sdc1 /dev/sdf1
* failpv(s):          /dev/sdb1 /dev/sdc1
* failnode(s):        taft-01
* additional snap:    /dev/sde1
* raid fault policy:   warn
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid1 -m 3 -n synced_multiple_raid1_3legs_1 -L 500M black_bird /dev/sde1:0-1000 /dev/sdb1:0-1000 /dev/sdc1:0-1000 /dev/sdf1:0-1000

RAID Structure(s):
  LV                                       Attr     LSize   Copy%  Devices
  synced_multiple_raid1_3legs_1            rwi-a-m- 500.00m  12.00 synced_multiple_raid1_3legs_1_rimage_0(0),synced_multiple_raid1_3legs_1_rimage_1(0),synced_multiple_raid1_3legs_1_rimage_2(0),synced_multiple_raid1_3legs_1_rimage_3(0)
  [synced_multiple_raid1_3legs_1_rimage_0] Iwi-aor- 500.00m        /dev/sde1(1)
  [synced_multiple_raid1_3legs_1_rimage_1] Iwi-aor- 500.00m        /dev/sdb1(1)
  [synced_multiple_raid1_3legs_1_rimage_2] Iwi-aor- 500.00m        /dev/sdc1(1)
  [synced_multiple_raid1_3legs_1_rimage_3] Iwi-aor- 500.00m        /dev/sdf1(1)
  [synced_multiple_raid1_3legs_1_rmeta_0]  ewi-aor-   4.00m        /dev/sde1(0)
  [synced_multiple_raid1_3legs_1_rmeta_1]  ewi-aor-   4.00m        /dev/sdb1(0)
  [synced_multiple_raid1_3legs_1_rmeta_2]  ewi-aor-   4.00m        /dev/sdc1(0)
  [synced_multiple_raid1_3legs_1_rmeta_3]  ewi-aor-   4.00m        /dev/sdf1(0)

PV=/dev/sdb1
        synced_multiple_raid1_3legs_1_rimage_1: 2
        synced_multiple_raid1_3legs_1_rmeta_1: 2
PV=/dev/sdc1
        synced_multiple_raid1_3legs_1_rimage_2: 2
        synced_multiple_raid1_3legs_1_rmeta_2: 2

Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 86.92% )
   1/1 mirror(s) are fully synced: ( 100.00% )

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Creating a snapshot volume of each of the raids
Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----

Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Disabling device sdb on taft-01
Disabling device sdc on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.488029 s, 85.9 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
  /dev/sdb1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 145669554176: Input/output error
  Couldn't find device with uuid IndNXV-cv33-7K3t-itAe-Kdz4-UtVl-hrMy6a.
  Couldn't find device with uuid 1HEmoV-uNl9-J0I7-XNnL-3zlZ-S2p8-9pwwQs.
  LV                                       Attr     LSize   Copy%  Devices
  bb_snap1                                 swi-a-s- 252.00m        /dev/sde1(126)
  synced_multiple_raid1_3legs_1            owi-aom- 500.00m 100.00 synced_multiple_raid1_3legs_1_rimage_0(0),synced_multiple_raid1_3legs_1_rimage_1(0),synced_multiple_raid1_3legs_1_rimage_2(0),synced_multiple_raid1_3legs_1_rimage_3(0)
  [synced_multiple_raid1_3legs_1_rimage_0] iwi-aor- 500.00m        /dev/sde1(1)
  [synced_multiple_raid1_3legs_1_rimage_1] iwi-aor- 500.00m        unknown device(1)
  [synced_multiple_raid1_3legs_1_rimage_2] iwi-aor- 500.00m        unknown device(1)
  [synced_multiple_raid1_3legs_1_rimage_3] iwi-aor- 500.00m        /dev/sdf1(1)
  [synced_multiple_raid1_3legs_1_rmeta_0]  ewi-aor-   4.00m        /dev/sde1(0)
  [synced_multiple_raid1_3legs_1_rmeta_1]  ewi-aor-   4.00m        unknown device(0)
  [synced_multiple_raid1_3legs_1_rmeta_2]  ewi-aor-   4.00m        unknown device(0)
  [synced_multiple_raid1_3legs_1_rmeta_3]  ewi-aor-   4.00m        /dev/sdf1(0)

Verifying FAILED device /dev/sdb1 is *NOT* in the volume(s)
Verifying FAILED device /dev/sdc1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sde1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdf1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_multiple_raid1_3legs_1_rimage_1 on:  taft-01
Checking EXISTENCE and STATE of synced_multiple_raid1_3legs_1_rmeta_1 on:  taft-01
Checking EXISTENCE and STATE of synced_multiple_raid1_3legs_1_rimage_2 on:  taft-01
Checking EXISTENCE and STATE of synced_multiple_raid1_3legs_1_rmeta_2 on:  taft-01

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sde1 unknown unknown /dev/sdf1
ACTUAL LEG ORDER: /dev/sde1 unknown unknown /dev/sdf1
Fault policy is warn, manually repairing failed raid volumes
taft-01: 'lvconvert --yes --repair black_bird/synced_multiple_raid1_3legs_1'

Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 65.76% )
   1/1 mirror(s) are fully synced: ( 100.00% )

Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Enabling device sdb on taft-01
Enabling device sdc on taft-01


Verify that each of the raid repairs finished successfully

Checking for leftover '-missing_0_0' or 'unknown devices'
there should no longer be any 'missing' dm devices on taft-01


[root@taft-01 ~]# dmsetup ls | grep missing
black_bird-synced_multiple_raid6_4legs_1_rmeta_0-missing_0_0    (253:20)
black_bird-synced_multiple_raid6_4legs_1_rimage_0-missing_0_0   (253:19)

Version-Release number of selected component (if applicable):
2.6.32-274.el6.x86_64
lvm2-2.02.95-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
lvm2-libs-2.02.95-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
lvm2-cluster-2.02.95-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
udev-147-2.41.el6    BUILT: Thu Mar  1 13:01:08 CST 2012
device-mapper-1.02.74-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
device-mapper-libs-1.02.74-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
device-mapper-event-1.02.74-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
device-mapper-event-libs-1.02.74-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012
cmirror-2.02.95-10.el6    BUILT: Fri May 18 03:26:00 CDT 2012


How reproducible:
Everytime
Comment 2 Corey Marthaler 2012-05-24 16:49:50 EDT
I posted the wrong test case information above. Closing this bug and opened 825026 with the correct test case output causing this problem.

Note You need to log in before you can comment on or make changes to this bug.