Bug 463520 - down convert no longer takes place after dev failure
down convert no longer takes place after dev failure
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: lvm2 (Show other bugs)
5.3
All Linux
medium Severity medium
: beta
: ---
Assigned To: Milan Broz
Cluster QE
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-09-23 16:06 EDT by Corey Marthaler
Modified: 2013-02-28 23:06 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 16:46:42 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2008-09-23 16:06:26 EDT
Description of problem:
This appears to be a pretty major regression. The mirror down convert no longer occurs after a device failure.

================================================================================
Iteration 0.1 started at Tue Sep 23 14:37:58 CDT 2008                           
================================================================================
Scenario: Kill primary leg of synced 2 leg mirror(s)                            

****** Mirror hash info for this scenario ******
* name:         syncd_primary_2legs             
* sync:         1                               
* num mirrors:  1                               
* disklog:      /dev/sde1                       
* failpv:       /dev/sdg1                       
* leg devices:  /dev/sdg1 /dev/sdf1             
************************************************

Creating mirror(s) on taft-02...
taft-02: lvcreate -m 1 -n syncd_primary_2legs_1 -L 600M helter_skelter /dev/sdg1:0-1000 /dev/sdf1:0-1000 /dev/sde1:0-150

Waiting until all mirrors become fully syncd...
        0/1 mirror(s) are fully synced: ( 1=1.00% )
        0/1 mirror(s) are fully synced: ( 1=51.42% )
        1/1 mirror(s) are fully synced: ( 1=100.00% )

Creating gfs on top of mirror(s) on taft-01...
Mounting mirrored gfs filesystems on taft-01...
Mounting mirrored gfs filesystems on taft-02...
Mounting mirrored gfs filesystems on taft-03...
Mounting mirrored gfs filesystems on taft-04...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----   
        ---- taft-02 ----
        ---- taft-03 ----
        ---- taft-04 ----

<start name="taft-01_1" pid="5590" time="Tue Sep 23 14:38:42 2008" type="cmd" />
<start name="taft-02_1" pid="5592" time="Tue Sep 23 14:38:42 2008" type="cmd" />
<start name="taft-03_1" pid="5594" time="Tue Sep 23 14:38:42 2008" type="cmd" />
<start name="taft-04_1" pid="5596" time="Tue Sep 23 14:38:42 2008" type="cmd" />
Sleeping 10 seconds to get some outsanding GFS I/O locks before the failure     
Verifying files (checkit) on mirror(s) on...                                    
        ---- taft-01 ----    
        ---- taft-02 ----
        ---- taft-03 ----
        ---- taft-04 ----

Disabling device sdg on taft-01
Disabling device sdg on taft-02
Disabling device sdg on taft-03
Disabling device sdg on taft-04

Attempting I/O to cause mirror down conversion(s) on taft-02
10+0 records in                                             
10+0 records out                                            
41943040 bytes (42 MB) copied, 6.12083 seconds, 6.9 MB/s    
Verifying the down conversion of the failed mirror(s)       
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629080064: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629137408: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error        
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 4096: Input/output error     
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error        
  /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error                                              
  /dev/sdg1: read failed after 0 of 512 at 145669664768: Input/output error                                              
  /dev/sdg1: read failed after 0 of 512 at 0: Input/output error                                                         
  /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error                                                      
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error                                                        
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.                                               
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error        
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error                                                        
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.                                               
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error        
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error                                                        
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.                                               
Verifying FAILED device /dev/sdg1 is *NOT* in the volume(s)                                                              
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error        
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629080064: Input/output e
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629137408: Input/output e
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.
Verifying LOG device /dev/sde1 is *NOT* in the linear(s)
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629080064: Input/output e
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 629137408: Input/output e
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.
  /dev/mapper/helter_skelter-syncd_primary_2legs_1_mimage_0: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 's0VFJf-Hvc3-vGo4-9tnH-d7Ec-NRRH-qUyqcL'.
log device /dev/sde1 should no longer be present on taft-02
run_iteration() method failed at /home/msp/cmarthal/work/rhel5/sts-root/lib/FI_engine.pm line 21.


Version-Release number of selected component (if applicable):
2.6.18-116.el5

lvm2-2.02.40-2.el5    BUILT: Fri Sep 19 09:46:26 CDT 2008
lvm2-cluster-2.02.40-2.el5    BUILT: Fri Sep 19 09:49:59 CDT 2008
device-mapper-1.02.28-2.el5    BUILT: Fri Sep 19 02:50:32 CDT 2008
cmirror-1.1.25-1.el5    BUILT: Fri Sep 19 16:27:46 CDT 2008
kmod-cmirror-0.1.17-1.el5    BUILT: Fri Sep 19 16:27:33 CDT 2008
openais-0.80.3-19.el5       BUILT: Tue 23 Sep 2008 12:58:51 PM CDT

How reproducible:
quite often

Steps to Reproduce:
Run helter_skelter's first scenario
Comment 1 Corey Marthaler 2008-09-24 17:44:54 EDT
This is reproducible on single machine mirrors as well.
Comment 2 Kiersten (Kerri) Anderson 2008-09-25 09:40:30 EDT
This should be a beta blocker - so setting request flag and devel ack'ing.
Comment 6 Corey Marthaler 2008-09-25 17:09:00 EDT
Fix verified in lvm2-2.02.40-3.el5/lvm2-cluster-2.02.40-3.el5.
Comment 8 errata-xmlrpc 2009-01-20 16:46:42 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0179.html

Note You need to log in before you can comment on or make changes to this bug.