Bug 471291 - RHEL5 cmirror tracker: log failure (on 3-way mirror) + node recovery results in corrupted filesystem
RHEL5 cmirror tracker: log failure (on 3-way mirror) + node recovery results ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: cmirror (Show other bugs)
5.3
All Linux
high Severity high
: rc
: ---
Assigned To: Jonathan Earl Brassow
Cluster QE
: Tracking
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-12 16:09 EST by Corey Marthaler
Modified: 2011-01-13 17:49 EST (History)
11 users (show)

See Also:
Fixed In Version: cmirror-1.1.39-9.el5
Doc Type: Bug Fix
Doc Text:
A data corruption may have occurred when using 3 or more mirrors. With this update, the underlying cluster code has been modified to address this issue, and the data corruption no longer occurs.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-01-13 17:49:00 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2008-11-12 16:09:29 EST
Description of problem:
I ran an iteration of helter_skelter on taft-0[1234] which killed the log device on a 3-way mirror and right after the log failure I killed taft-01 (the node with the lowest id). This resulted in an apparent corrupted filesystem.

It's possible that this may be the same issue as either bz 446255 or bz 460317 (both currently Modified).

Scenario: Kill disk log of synced 3 leg mirror(s)                               

****** Mirror hash info for this scenario ******
* name:         syncd_log_3legs                 
* sync:         1                               
* num mirrors:  1                               
* disklog:      /dev/sde1                       
* failpv:       /dev/sde1                       
* leg devices:  /dev/sdh1 /dev/sdf1 /dev/sdg1   
************************************************

Creating mirror(s) on taft-02...
taft-02: lvcreate -m 2 -n syncd_log_3legs_1 -L 600M helter_skelter /dev/sdh1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sde1:0-150                                                                                           

Waiting until all mirrors become fully syncd...
        0/1 mirror(s) are fully synced: ( 1=1.25% )
        0/1 mirror(s) are fully synced: ( 1=46.58% )
        0/1 mirror(s) are fully synced: ( 1=91.08% )
        1/1 mirror(s) are fully synced: ( 1=100.00% )

Creating gfs on top of mirror(s) on taft-01...
Mounting mirrored gfs filesystems on taft-01...
Mounting mirrored gfs filesystems on taft-02...
Mounting mirrored gfs filesystems on taft-03...
Mounting mirrored gfs filesystems on taft-04...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----   
        ---- taft-02 ----
        ---- taft-03 ----
        ---- taft-04 ----

<start name="taft-01_1" pid="9006" time="Wed Nov 12 14:20:27 2008" type="cmd" />
<start name="taft-02_1" pid="9008" time="Wed Nov 12 14:20:27 2008" type="cmd" />
<start name="taft-03_1" pid="9010" time="Wed Nov 12 14:20:27 2008" type="cmd" />
<start name="taft-04_1" pid="9012" time="Wed Nov 12 14:20:27 2008" type="cmd" />
Sleeping 10 seconds to get some outsanding GFS I/O locks before the failure     

Verifying files (checkit) on mirror(s) on...                                    
        ---- taft-01 ----           
        ---- taft-02 ----
        ---- taft-03 ----
        ---- taft-04 ----

Disabling device sde on taft-01
Disabling device sde on taft-02
Disabling device sde on taft-03
Disabling device sde on taft-04

Attempting I/O to cause mirror down conversion(s) on taft-02
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 23.2351 seconds, 1.8 MB/s
Verifying the down conversion of the failed mirror(s)
  /dev/sde1: open failed: No such device or address
Verifying FAILED device /dev/sde1 is *NOT* in the volume(s)
  /dev/sde1: open failed: No such device or address
Verifying LEG device /dev/sdh1 *IS* in the volume(s)
  /dev/sde1: open failed: No such device or address
Verifying LEG device /dev/sdf1 *IS* in the volume(s)
  /dev/sde1: open failed: No such device or address
Verifying LEG device /dev/sdg1 *IS* in the volume(s)
  /dev/sde1: open failed: No such device or address
Verify the dm devices associated with /dev/sde1 are no longer present
<fail name="taft-01_1" pid="9006" time="Wed Nov 12 14:23:08 2008" type="cmd" duration="161" ec="127" />
ALL STOP!
Verify the order of legs didn't shift after the down conversion
  /dev/sde1: open failed: No such device or address
  /dev/sde1: open failed: No such device or address
  /dev/sde1: open failed: No such device or address

[ HERE I KILLED TAFT-01 ]

From taft-02:
Nov 12 14:21:05 taft-02 kernel: GFS: fsid=TAFT:gfs1.1: jid=0: Looking at journal...
Nov 12 14:21:05 taft-02 kernel: GFS: fsid=TAFT:gfs1.1: jid=0: Acquiring the transaction lock...
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1: jid=0: Replaying journal...
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1: fatal: invalid metadata block
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1:   bh = 4462 (magic)
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1:   function = replay_block
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1:   file = /builddir/build/BUILD/gfs-kmod-0.1.24/_kmod_build_/src/gfs/lops.c, line = 548
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1:   time = 1226521266
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1: about to withdraw from the cluster
Nov 12 14:21:06 taft-02 kernel: GFS: fsid=TAFT:gfs1.1: telling LM to withdraw
Nov 12 14:21:21 taft-02 kernel: device-mapper: dm-log-clustered: [HK7uR6Sr] Request timed out: [DM_CLOG_FLUSH/2320964] - retrying

From a simple straced touch:
write(2, ": Input/output error", 20: Input/output error)    = 20

The sync process is now stuck as well:
[root@taft-02 sbin]# lvs -a -o +devices
  /dev/sde1: open failed: No such device or address
  LV                           VG             Attr   LSize   Origin Snap%  Move Log Copy%  Convert Devices                                                                                  
  LogVol00                     VolGroup00     -wi-ao  66.19G                                       /dev/sda2(0)                                                                             
  LogVol01                     VolGroup00     -wi-ao   1.94G                                       /dev/sda2(2118)                                                                          
  syncd_log_3legs_1            helter_skelter mwi-so 600.00M                         99.33         syncd_log_3legs_1_mimage_0(0),syncd_log_3legs_1_mimage_1(0),syncd_log_3legs_1_mimage_2(0)
  [syncd_log_3legs_1_mimage_0] helter_skelter Iwi-ao 600.00M                                       /dev/sdh1(0)                                                                             
  [syncd_log_3legs_1_mimage_1] helter_skelter Iwi-ao 600.00M                                       /dev/sdf1(0)                                                                             
  [syncd_log_3legs_1_mimage_2] helter_skelter Iwi-ao 600.00M                                       /dev/sdg1(0)

Version-Release number of selected component (if applicable):
2.6.18-117.el5

lvm2-2.02.40-6.el5    BUILT: Fri Oct 24 07:37:33 CDT 2008
lvm2-cluster-2.02.40-6.el5    BUILT: Fri Oct 24 07:38:44 CDT 2008
device-mapper-1.02.28-2.el5    BUILT: Fri Sep 19 02:50:32 CDT 2008
cmirror-1.1.34-5.el5    BUILT: Thu Nov  6 15:10:44 CST 2008
kmod-cmirror-0.1.21-2.el5    BUILT: Thu Nov  6 14:12:07 CST 2008
Comment 4 Corey Marthaler 2008-12-02 14:53:53 EST
QA is currently unable to reproduce this bug on 2-way mirrors, so it looks like this *is* only 3-way mirror related. That, or I'm getting the node recovery timing wrong. Until I prove otherwise, I'm changing the subject to reflect this.
Comment 7 RHEL Product and Program Management 2009-11-06 13:55:48 EST
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".
Comment 9 RHEL Product and Program Management 2010-08-30 16:36:52 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 11 Corey Marthaler 2010-11-03 17:17:22 EDT
I've run this scenario a few times now with the latest packages and haven't seen this issue. Marking verified.

2.6.18-227.el5

lvm2-2.02.74-1.el5    BUILT: Fri Oct 15 10:26:21 CDT 2010
lvm2-cluster-2.02.74-2.el5    BUILT: Fri Oct 29 07:48:11 CDT 2010
device-mapper-1.02.55-1.el5    BUILT: Fri Oct 15 06:15:55 CDT 2010
cmirror-1.1.39-10.el5    BUILT: Wed Sep  8 16:32:05 CDT 2010
kmod-cmirror-0.1.22-3.el5    BUILT: Tue Dec 22 13:39:47 CST 2009
Comment 12 Jaromir Hradilek 2010-11-17 09:29:41 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
A data corruption may have occurred when using 3 or more mirrors. With this update, the underlying cluster code has been modified to address this issue, and the data corruption no longer occurs.
Comment 14 errata-xmlrpc 2011-01-13 17:49:00 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0057.html

Note You need to log in before you can comment on or make changes to this bug.