Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 607316 - primary log device removal after failure does not work
primary log device removal after failure does not work
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.0
All Linux
high Severity high
: rc
: ---
Assigned To: Petr Rockai
Corey Marthaler
: TestBlocker
Depends On: 606931
Blocks:
  Show dependency treegraph
 
Reported: 2010-06-23 15:22 EDT by Corey Marthaler
Modified: 2010-11-10 16:08 EST (History)
12 users (show)

See Also:
Fixed In Version: lvm2-2.02.72-7.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-10 16:08:10 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
cmarthal: needinfo+


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2010-06-23 15:22:57 EDT
Description of problem:
Scenario: Kill primary log of synced 2 leg redundand log mirror(s)

********* Mirror hash info for this scenario *********
* names:              syncd_primary_log_2legs_2logs_1 
* sync:               1                               
* leg devices:        /dev/sdg1 /dev/sdb1             
* log devices:        /dev/sdh1 /dev/sdd1             
* failpv(s):          /dev/sdh1                       
* failnode(s):        taft-01                         
* leg fault policy:   remove                          
* log fault policy:   allocate                        
******************************************************

Creating mirror(s) on taft-01...
taft-01: lvcreate --mirrorlog mirrored -m 1 -n syncd_primary_log_2legs_2logs_1 -L 600M helter_skelter /dev/sdg1:0-1000 /dev/sdb1:0-1000 /dev/sdh1:0-150 /dev/sdd1:0-150

PV=/dev/sdh1
        syncd_primary_log_2legs_2logs_1_mlog_mimage_0: 1.2
PV=/dev/sdh1                                              
        syncd_primary_log_2legs_2logs_1_mlog_mimage_0: 1.2

Waiting until all mirrors become fully syncd...
   0/1 mirror(s) are fully synced: ( 63.25% )  
   1/1 mirror(s) are fully synced: ( 100.00% ) 

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.10 (10-Feb-2009)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----                              

Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Disabling device sdh on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.240536 s, 174 MB/s
Verifying current sanity of lvm after the failure
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
  [...]
  Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v.
Verifying FAILED device /dev/sdh1 is *NOT* in the volume(s)
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
  [...]
  Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v.
Verifying LOG device(s) /dev/sdh1 /dev/sdd1 are *NOT* in the linear(s)
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
  [...]
  Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v.
log device /dev/sdd1 should no longer be present on taft-01


# mirror before the failure
syncd_primary_log_2legs_2logs_1                  helter_skelter mwi-ao 600.00m  syncd_primary_log_2legs_2logs_1_mlog 100.00 syncd_primary_log_2legs_2logs_1_mimage_0(0),syncd_primary_log_2legs_2logs_1_mimage_1(0)
[syncd_primary_log_2legs_2logs_1_mimage_0]       helter_skelter iwi-ao 600.00m                                              /dev/sdg1(0)
[syncd_primary_log_2legs_2logs_1_mimage_1]       helter_skelter iwi-ao 600.00m                                              /dev/sdb1(0)
[syncd_primary_log_2legs_2logs_1_mlog]           helter_skelter mwi-ao   4.00m                                       100.00 syncd_primary_log_2legs_2logs_1_mlog_mimage_0(0),syncd_primary_log_2legs_2logs_1_mlog_mimage_1(0)
[syncd_primary_log_2legs_2logs_1_mlog_mimage_0]  helter_skelter iwi-ao   4.00m                                              /dev/sdh1(0)
[syncd_primary_log_2legs_2logs_1_mlog_mimage_1]  helter_skelter iwi-ao   4.00m                                              /dev/sdd1(0)

# mirror after the failure
syncd_primary_log_2legs_2logs_1               helter_skelter mwi-ao 600.00m  syncd_primary_log_2legs_2logs_1_mlog 100.00  syncd_primary_log_2legs_2logs_1_mimage_0(0),syncd_primary_log_2legs_2logs_1_mimage_1(0)
[syncd_primary_log_2legs_2logs_1_mimage_0]    helter_skelter iwi-ao 600.00m                                               /dev/sdg1(0)
[syncd_primary_log_2legs_2logs_1_mimage_1]    helter_skelter iwi-ao 600.00m                                               /dev/sdb1(0)
[syncd_primary_log_2legs_2logs_1_mlog]        helter_skelter lwi-ao   4.00m                                               /dev/sdd1(0)
syncd_primary_log_2legs_2logs_1_mlog_mimage_0 helter_skelter -wi-a-   4.00m                                               unknown device(0)
syncd_primary_log_2legs_2logs_1_mlog_mimage_1 helter_skelter vwi-a-   4.00m

# console
Jun 23 18:59:02 taft-01 kernel: sd 3:0:0:7: rejecting I/O to offline device
Jun 23 18:59:02 taft-01 lvm[1950]: LV helter_skelter/syncd_primary_log_2legs_2logs_1_mlog_mimage_0 in use: not deactivating
Jun 23 18:59:02 taft-01 lvm[1950]: Repair of mirrored LV helter_skelter/syncd_primary_log_2legs_2logs_1 failed.
Jun 23 18:59:02 taft-01 lvm[1950]: Failed to remove faulty devices in helter_skelter-syncd_primary_log_2legs_2logs_1_mlog.
Jun 23 18:59:02 taft-01 kernel: sd 3:0:0:7: rejecting I/O to offline device

Version-Release number of selected component (if applicable):
Version 2.02.67+ (custom build)
Comment 1 Petr Rockai 2010-06-29 19:00:38 EDT
For all I can tell, this is due to incomplete support for mirrored log in lvconvert --repair. I am working on it.
Comment 2 Petr Rockai 2010-06-29 19:06:25 EDT
I take that back, I have noticed this:

Jun 23 18:59:02 taft-01 lvm[1950]: LV helter_skelter/syncd_primary_log_2legs_2logs_1_mlog_mimage_0 in use: not deactivating

this looks related to bug 606931. Let's mark this as a dependency -- when 606931 is fixed we should see whether this got fixed as well, or there is an additional issue specific to mirrored logs.
Comment 4 Petr Rockai 2010-08-09 11:06:36 EDT
Relevant patch checked in upstream.
Comment 5 Corey Marthaler 2010-08-12 16:35:03 EDT
Is this fix in the latest build (lvm2-2.02.72-7.el6.x86_64)? If so, a new log device is not allocated, instead the mirror is down convert to a linear. Is this what's expected?
Comment 6 Petr Rockai 2010-08-13 06:32:50 EDT
Hmm. Do you mean that the whole mirror is downconverted, or just that the mirrored log is downconverted to linear? If the former happens, that is certainly surprising, since the fallback policy on that scenario is to use core log which should certainly not fail if both (toplevel) mirror legs are healthy.

On the other hand, if it is the mirrored log that ends up linear, then that may be unintentional but is probably not a serious problem (we can fix this in 6.1 in that case).

Certainly detailed output would help in any case to clarify what's going on. Thanks.
Comment 7 Corey Marthaler 2010-08-17 14:51:10 EDT
It's the mirrored log that ends up a linear.

image0 sdb
image1 sdc
log image0 sdd
log image1 sde

if you fail sdd and your fault policy is allocate, the mirror will end up looking like this:

image0 sdb
image1 sdc
log sde

Jon explained to me that this is expected (at least for the time being) since there is no way to go from disk to mirrored log with the current code.
Comment 12 Corey Marthaler 2010-08-18 16:32:34 EDT
The original issue in this bug is fixed. The issue mentioned in comment #7 is documented in bug 607334. 

Marking verified in the latest rpms.

2.6.32-59.1.el6.x86_64

lvm2-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
lvm2-libs-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
lvm2-cluster-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
udev-147-2.22.el6    BUILT: Fri Jul 23 07:21:33 CDT 2010
device-mapper-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
device-mapper-libs-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
device-mapper-event-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
device-mapper-event-libs-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
cmirror-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
Comment 13 releng-rhel@redhat.com 2010-11-10 16:08:10 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.