RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 607316 - primary log device removal after failure does not work
Summary: primary log device removal after failure does not work
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Petr Rockai
QA Contact: Corey Marthaler
URL:
Whiteboard:
Depends On: 606931
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-06-23 19:22 UTC by Corey Marthaler
Modified: 2010-11-10 21:08 UTC (History)
12 users (show)

Fixed In Version: lvm2-2.02.72-7.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-11-10 21:08:10 UTC
Target Upstream Version:
Embargoed:
cmarthal: needinfo+

Attachments (Terms of Use)

Description Corey Marthaler 2010-06-23 19:22:57 UTC 
Description of problem:
Scenario: Kill primary log of synced 2 leg redundand log mirror(s)

********* Mirror hash info for this scenario *********
* names:              syncd_primary_log_2legs_2logs_1 
* sync:               1                               
* leg devices:        /dev/sdg1 /dev/sdb1             
* log devices:        /dev/sdh1 /dev/sdd1             
* failpv(s):          /dev/sdh1                       
* failnode(s):        taft-01                         
* leg fault policy:   remove                          
* log fault policy:   allocate                        
******************************************************

Creating mirror(s) on taft-01...
taft-01: lvcreate --mirrorlog mirrored -m 1 -n syncd_primary_log_2legs_2logs_1 -L 600M helter_skelter /dev/sdg1:0-1000 /dev/sdb1:0-1000 /dev/sdh1:0-150 /dev/sdd1:0-150

PV=/dev/sdh1
        syncd_primary_log_2legs_2logs_1_mlog_mimage_0: 1.2
PV=/dev/sdh1                                              
        syncd_primary_log_2legs_2logs_1_mlog_mimage_0: 1.2

Waiting until all mirrors become fully syncd...
   0/1 mirror(s) are fully synced: ( 63.25% )  
   1/1 mirror(s) are fully synced: ( 100.00% ) 

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.10 (10-Feb-2009)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----                              

Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Disabling device sdh on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.240536 s, 174 MB/s
Verifying current sanity of lvm after the failure
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
  [...]
  Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v.
Verifying FAILED device /dev/sdh1 is *NOT* in the volume(s)
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
  [...]
  Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v.
Verifying LOG device(s) /dev/sdh1 /dev/sdd1 are *NOT* in the linear(s)
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
  [...]
  Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v.
log device /dev/sdd1 should no longer be present on taft-01


# mirror before the failure
syncd_primary_log_2legs_2logs_1                  helter_skelter mwi-ao 600.00m  syncd_primary_log_2legs_2logs_1_mlog 100.00 syncd_primary_log_2legs_2logs_1_mimage_0(0),syncd_primary_log_2legs_2logs_1_mimage_1(0)
[syncd_primary_log_2legs_2logs_1_mimage_0]       helter_skelter iwi-ao 600.00m                                              /dev/sdg1(0)
[syncd_primary_log_2legs_2logs_1_mimage_1]       helter_skelter iwi-ao 600.00m                                              /dev/sdb1(0)
[syncd_primary_log_2legs_2logs_1_mlog]           helter_skelter mwi-ao   4.00m                                       100.00 syncd_primary_log_2legs_2logs_1_mlog_mimage_0(0),syncd_primary_log_2legs_2logs_1_mlog_mimage_1(0)
[syncd_primary_log_2legs_2logs_1_mlog_mimage_0]  helter_skelter iwi-ao   4.00m                                              /dev/sdh1(0)
[syncd_primary_log_2legs_2logs_1_mlog_mimage_1]  helter_skelter iwi-ao   4.00m                                              /dev/sdd1(0)

# mirror after the failure
syncd_primary_log_2legs_2logs_1               helter_skelter mwi-ao 600.00m  syncd_primary_log_2legs_2logs_1_mlog 100.00  syncd_primary_log_2legs_2logs_1_mimage_0(0),syncd_primary_log_2legs_2logs_1_mimage_1(0)
[syncd_primary_log_2legs_2logs_1_mimage_0]    helter_skelter iwi-ao 600.00m                                               /dev/sdg1(0)
[syncd_primary_log_2legs_2logs_1_mimage_1]    helter_skelter iwi-ao 600.00m                                               /dev/sdb1(0)
[syncd_primary_log_2legs_2logs_1_mlog]        helter_skelter lwi-ao   4.00m                                               /dev/sdd1(0)
syncd_primary_log_2legs_2logs_1_mlog_mimage_0 helter_skelter -wi-a-   4.00m                                               unknown device(0)
syncd_primary_log_2legs_2logs_1_mlog_mimage_1 helter_skelter vwi-a-   4.00m

# console
Jun 23 18:59:02 taft-01 kernel: sd 3:0:0:7: rejecting I/O to offline device
Jun 23 18:59:02 taft-01 lvm[1950]: LV helter_skelter/syncd_primary_log_2legs_2logs_1_mlog_mimage_0 in use: not deactivating
Jun 23 18:59:02 taft-01 lvm[1950]: Repair of mirrored LV helter_skelter/syncd_primary_log_2legs_2logs_1 failed.
Jun 23 18:59:02 taft-01 lvm[1950]: Failed to remove faulty devices in helter_skelter-syncd_primary_log_2legs_2logs_1_mlog.
Jun 23 18:59:02 taft-01 kernel: sd 3:0:0:7: rejecting I/O to offline device

Version-Release number of selected component (if applicable):
Version 2.02.67+ (custom build)
Comment 1 Petr Rockai 2010-06-29 23:00:38 UTC 
For all I can tell, this is due to incomplete support for mirrored log in lvconvert --repair. I am working on it.
Comment 2 Petr Rockai 2010-06-29 23:06:25 UTC 
I take that back, I have noticed this:

Jun 23 18:59:02 taft-01 lvm[1950]: LV helter_skelter/syncd_primary_log_2legs_2logs_1_mlog_mimage_0 in use: not deactivating

this looks related to bug 606931. Let's mark this as a dependency -- when 606931 is fixed we should see whether this got fixed as well, or there is an additional issue specific to mirrored logs.
Comment 4 Petr Rockai 2010-08-09 15:06:36 UTC 
Relevant patch checked in upstream.
Comment 5 Corey Marthaler 2010-08-12 20:35:03 UTC 
Is this fix in the latest build (lvm2-2.02.72-7.el6.x86_64)? If so, a new log device is not allocated, instead the mirror is down convert to a linear. Is this what's expected?
Comment 6 Petr Rockai 2010-08-13 10:32:50 UTC 
Hmm. Do you mean that the whole mirror is downconverted, or just that the mirrored log is downconverted to linear? If the former happens, that is certainly surprising, since the fallback policy on that scenario is to use core log which should certainly not fail if both (toplevel) mirror legs are healthy.

On the other hand, if it is the mirrored log that ends up linear, then that may be unintentional but is probably not a serious problem (we can fix this in 6.1 in that case).

Certainly detailed output would help in any case to clarify what's going on. Thanks.
Comment 7 Corey Marthaler 2010-08-17 18:51:10 UTC 
It's the mirrored log that ends up a linear.

image0 sdb
image1 sdc
log image0 sdd
log image1 sde

if you fail sdd and your fault policy is allocate, the mirror will end up looking like this:

image0 sdb
image1 sdc
log sde

Jon explained to me that this is expected (at least for the time being) since there is no way to go from disk to mirrored log with the current code.
Comment 12 Corey Marthaler 2010-08-18 20:32:34 UTC 
The original issue in this bug is fixed. The issue mentioned in comment #7 is documented in bug 607334. 

Marking verified in the latest rpms.

2.6.32-59.1.el6.x86_64

lvm2-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
lvm2-libs-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
lvm2-cluster-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
udev-147-2.22.el6    BUILT: Fri Jul 23 07:21:33 CDT 2010
device-mapper-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
device-mapper-libs-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
device-mapper-event-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
device-mapper-event-libs-1.02.53-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
cmirror-2.02.72-8.el6    BUILT: Wed Aug 18 10:41:52 CDT 2010
Comment 13 releng-rhel@redhat.com 2010-11-10 21:08:10 UTC 
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.
Note You need to log in before you can comment on or make changes to this bug.