Bug 758552
Summary: | RFE: LVM RAID - Handle device failures | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Jonathan Earl Brassow <jbrassow> |
Component: | lvm2 | Assignee: | Jonathan Earl Brassow <jbrassow> |
Status: | CLOSED ERRATA | QA Contact: | Cluster QE <mspqa-list> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 6.2 | CC: | agk, cmarthal, dwysocha, heinzm, jbrassow, mbroz, prajnoha, prockai, thornber, zkabelac |
Target Milestone: | rc | Keywords: | FutureFeature |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | lvm2-2.02.95-1.el6 | Doc Type: | Enhancement |
Doc Text: |
New Feature to 6.3. No documentation required.
Bug 732458 is the bug that requires a release note for the RAID features. Other documentation is found in the LVM manual.
Operational bugs need no documentation because they are being fixed before their initial release.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2012-06-20 15:00:28 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 593119, 732458 |
Description
Jonathan Earl Brassow
2011-11-30 02:38:04 UTC
LVM's configuration file (lvm.conf) now contains a field called 'raid_fault_policy', which determines the automated action taken in the event of a device failure. The options for this field are: "warn" - produce a warning "allocate" - Attempt to replace the failed device with a spare found in the VG The release criteria is that these two options work as expected. That is, if the policy is set to "warn", you should find notices in the log that a device has failed (and preferably, what to do about it). If the policy is set to "allocate", then the failed device should be replaced by a spare if there is one and if not, report the failure to replace the device in the system log. This feature has been added to LVM version 2.02.89. # Automated recovery in action (raid_fault_policy = "allocate") [root@bp-01 ~]# devices vg LV Copy% Devices lv 100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0) [lv_rimage_0] /dev/sde1(1) [lv_rimage_1] /dev/sdf1(1) [lv_rimage_2] /dev/sdg1(1) [lv_rmeta_0] /dev/sde1(0) [lv_rmeta_1] /dev/sdf1(0) [lv_rmeta_2] /dev/sdg1(0) [root@bp-01 ~]# grep raid_fau //etc/lvm/lvm.conf raid_fault_policy = "allocate" # kill device [root@bp-01 ~]# off.sh sde Turning off sde #write to LV [root@bp-01 ~]# dd if=/dev/zero of=/dev/vg/lv bs=4M count=1 1+0 records in 1+0 records out 4194304 bytes (4.2 MB) copied, 0.191669 s, 21.9 MB/s # system log doesn't have details of fix - this is probably worth a new bug! [root@bp-01 ~]# grep lvm /var/log/messages Jan 17 15:57:18 bp-01 lvm[8599]: Device #0 of raid1 array, vg-lv, has failed. Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at 250994294784: Input/output error Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at 250994376704: Input/output error Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at 0: Input/output error Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at 4096: Input/output error Jan 17 15:57:19 bp-01 lvm[8599]: Couldn't find device with uuid 3lugiV-3eSP-AFAR-sdrP-H20O-wM2M-qdMANy. Jan 17 15:57:27 bp-01 lvm[8599]: raid1 array, vg-lv, is not in-sync. Jan 17 15:57:36 bp-01 lvm[8599]: raid1 array, vg-lv, is now in-sync. # Notice the device has been replaced with a new one [root@bp-01 ~]# devices Couldn't find device with uuid 3lugiV-3eSP-AFAR-sdrP-H20O-wM2M-qdMANy. LV Copy% Devices lv 100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0) [lv_rimage_0] /dev/sdh1(1) [lv_rimage_1] /dev/sdf1(1) [lv_rimage_2] /dev/sdg1(1) [lv_rmeta_0] /dev/sdh1(0) [lv_rmeta_1] /dev/sdf1(0) [lv_rmeta_2] /dev/sdg1(0) lv_home /dev/sda2(12800) lv_root /dev/sda2(0) lv_swap /dev/sda2(35493) # Manual handling of failures [root@bp-01 ~]# devices vg LV Copy% Devices lv 100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0) [lv_rimage_0] /dev/sdh1(1) [lv_rimage_1] /dev/sdf1(1) [lv_rimage_2] /dev/sdg1(1) [lv_rmeta_0] /dev/sdh1(0) [lv_rmeta_1] /dev/sdf1(0) [lv_rmeta_2] /dev/sdg1(0) #kill device [root@bp-01 ~]# off.sh sdh Turning off sdh # run repair by hand # Again note the scarce output signifying success - this is probably worth a new # bug [root@bp-01 ~]# lvconvert --repair vg/lv /dev/sdh1: read failed after 0 of 2048 at 250994294784: Input/output error /dev/sdh1: read failed after 0 of 2048 at 250994376704: Input/output error /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error /dev/sdh1: read failed after 0 of 2048 at 4096: Input/output error Couldn't find device with uuid fbI0YO-GX7x-firU-Vy5o-vzwx-vAKZ-feRxfF. Attempt to replace failed RAID images (requires full device resync)? [y/n]: y # Note the replaced device [root@bp-01 ~]# devices vg Couldn't find device with uuid fbI0YO-GX7x-firU-Vy5o-vzwx-vAKZ-feRxfF. LV Copy% Devices lv 64.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0) [lv_rimage_0] /dev/sde1(1) [lv_rimage_1] /dev/sdf1(1) [lv_rimage_2] /dev/sdg1(1) [lv_rmeta_0] /dev/sde1(0) [lv_rmeta_1] /dev/sdf1(0) [lv_rmeta_2] /dev/sdg1(0) The following raid device failure test cases now pass: kill_primary_synced_raid1_2legs kill_primary_synced_raid4_2legs kill_primary_synced_raid5_2legs kill_primary_synced_raid1_3legs kill_primary_synced_raid4_3legs kill_primary_synced_raid5_3legs kill_primary_synced_raid6_3legs kill_random_synced_raid1_2legs kill_random_synced_raid4_2legs kill_random_synced_raid5_2legs kill_random_synced_raid1_3legs kill_random_synced_raid4_3legs kill_random_synced_raid5_3legs kill_random_synced_raid6_3legs kill_primary_synced_raid6_4legs kill_random_synced_raid6_4legs kill_multiple_synced_raid1_3legs kill_multiple_synced_raid6_3legs kill_multiple_synced_raid1_4legs kill_primary_non_synced_raid1_2legs kill_primary_non_synced_raid1_3legs kill_random_non_synced_raid1_2legs kill_random_non_synced_raid1_3legs Marking this feature verified in the latest rpms. 2.6.32-262.el6.x86_64 lvm2-2.02.95-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 lvm2-libs-2.02.95-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 lvm2-cluster-2.02.95-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 udev-147-2.41.el6 BUILT: Thu Mar 1 13:01:08 CST 2012 device-mapper-1.02.74-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 device-mapper-libs-1.02.74-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 device-mapper-event-1.02.74-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 device-mapper-event-libs-1.02.74-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 cmirror-2.02.95-5.el6 BUILT: Thu Apr 19 10:29:01 CDT 2012 Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: New Feature to 6.3. No documentation required. Bug 732458 is the bug that requires a release note for the RAID features. Other documentation is found in the LVM manual. Operational bugs need no documentation because they are being fixed before their initial release. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2012-0962.html |