Description of problem: This appears to be a pretty major regression. The mirror down convert no longer occurs after a device failure. ================================================================================ Iteration 0.1 started at Tue Sep 23 14:37:58 CDT 2008 ================================================================================ Scenario: Kill primary leg of synced 2 leg mirror(s) ****** Mirror hash info for this scenario ****** * name: syncd_primary_2legs * sync: 1 * num mirrors: 1 * disklog: /dev/sde1 * failpv: /dev/sdg1 * leg devices: /dev/sdg1 /dev/sdf1 ************************************************ Creating mirror(s) on taft-02... taft-02: lvcreate -m 1 -n syncd_primary_2legs_1 -L 600M helter_skelter /dev/sdg1:0-1000 /dev/sdf1:0-1000 /dev/sde1:0-150 Waiting until all mirrors become fully syncd... 0/1 mirror(s) are fully synced: ( 1=1.00% ) 0/1 mirror(s) are fully synced: ( 1=51.42% ) 1/1 mirror(s) are fully synced: ( 1=100.00% ) Creating gfs on top of mirror(s) on taft-01... Mounting mirrored gfs filesystems on taft-01... Mounting mirrored gfs filesystems on taft-02... Mounting mirrored gfs filesystems on taft-03... Mounting mirrored gfs filesystems on taft-04... Writing verification files (checkit) to mirror(s) on... ---- taft-01 ---- ---- taft-02 ---- ---- taft-03 ---- ---- taft-04 ---- <start name="taft-01_1" pid="5590" time="Tue Sep 23 14:38:42 2008" type="cmd" /> <start name="taft-02_1" pid="5592" time="Tue Sep 23 14:38:42 2008" type="cmd" /> <start name="taft-03_1" pid="5594" time="Tue Sep 23 14:38:42 2008" type="cmd" /> <start name="taft-04_1" pid="5596" time="Tue Sep 23 14:38:42 2008" type="cmd" /> Sleeping 10 seconds to get some outsanding GFS I/O locks before the failure Verifying files (checkit) on mirror(s) on... ---- taft-01 ---- ---- taft-02 ---- ---- taft-03 ---- ---- taft-04 ---- Disabling device sdg on taft-01 Disabling device sdg on taft-02 Disabling device sdg on taft-03 Disabling device sdg on taft-04 Attempting I/O to cause mirror down conversion(s) on taft-02 10+0 records in 10+0 records out 41943040 bytes (42 MB) copied, 6.12083 seconds, 6.9 MB/s Verifying the down conversion of the failed mirror(s) /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629080064: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629137408: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 4096: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error /dev/sdg1: read failed after 0 of 512 at 145669664768: Input/output error /dev/sdg1: read failed after 0 of 512 at 0: Input/output error /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. Verifying FAILED device /dev/sdg1 is *NOT* in the volume(s) /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629080064: Input/output e /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629137408: Input/output e /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 4096: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error /dev/sdg1: read failed after 0 of 512 at 145669664768: Input/output error /dev/sdg1: read failed after 0 of 512 at 0: Input/output error /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. Verifying LOG device /dev/sde1 is *NOT* in the linear(s) /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629080064: Input/output e /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629137408: Input/output e /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 4096: Input/output error /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error /dev/sdg1: read failed after 0 of 512 at 145669664768: Input/output error /dev/sdg1: read failed after 0 of 512 at 0: Input/output error /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'. log device /dev/sde1 should no longer be present on taft-02 run_iteration() method failed at /home/msp/cmarthal/work/rhel5/sts-root/lib/FI_engine.pm line 21. Version-Release number of selected component (if applicable): 2.6.18-116.el5 lvm2-2.02.40-2.el5 BUILT: Fri Sep 19 09:46:26 CDT 2008 lvm2-cluster-2.02.40-2.el5 BUILT: Fri Sep 19 09:49:59 CDT 2008 device-mapper-1.02.28-2.el5 BUILT: Fri Sep 19 02:50:32 CDT 2008 cmirror-1.1.25-1.el5 BUILT: Fri Sep 19 16:27:46 CDT 2008 kmod-cmirror-0.1.17-1.el5 BUILT: Fri Sep 19 16:27:33 CDT 2008 openais-0.80.3-19.el5 BUILT: Tue 23 Sep 2008 12:58:51 PM CDT How reproducible: quite often Steps to Reproduce: Run helter_skelter's first scenario
This is reproducible on single machine mirrors as well.
This should be a beta blocker - so setting request flag and devel ack'ing.
Fix verified in lvm2-2.02.40-3.el5/lvm2-cluster-2.02.40-3.el5.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-0179.html