Bug 607316
| Summary: | primary log device removal after failure does not work | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Corey Marthaler <cmarthal> |
| Component: | lvm2 | Assignee: | Petr Rockai <prockai> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Corey Marthaler <cmarthal> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.0 | CC: | agk, antillon.maurizio, coughlan, ddumas, dwysocha, heinzm, jbrassow, joe.thornber, mbroz, prajnoha, prockai, syeghiay |
| Target Milestone: | rc | Keywords: | TestBlocker |
| Target Release: | --- | Flags: | cmarthal:
needinfo+
|
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | lvm2-2.02.72-7.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2010-11-10 21:08:10 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 606931 | ||
| Bug Blocks: | |||
For all I can tell, this is due to incomplete support for mirrored log in lvconvert --repair. I am working on it. I take that back, I have noticed this: Jun 23 18:59:02 taft-01 lvm[1950]: LV helter_skelter/syncd_primary_log_2legs_2logs_1_mlog_mimage_0 in use: not deactivating this looks related to bug 606931. Let's mark this as a dependency -- when 606931 is fixed we should see whether this got fixed as well, or there is an additional issue specific to mirrored logs. Relevant patch checked in upstream. Is this fix in the latest build (lvm2-2.02.72-7.el6.x86_64)? If so, a new log device is not allocated, instead the mirror is down convert to a linear. Is this what's expected? Hmm. Do you mean that the whole mirror is downconverted, or just that the mirrored log is downconverted to linear? If the former happens, that is certainly surprising, since the fallback policy on that scenario is to use core log which should certainly not fail if both (toplevel) mirror legs are healthy. On the other hand, if it is the mirrored log that ends up linear, then that may be unintentional but is probably not a serious problem (we can fix this in 6.1 in that case). Certainly detailed output would help in any case to clarify what's going on. Thanks. It's the mirrored log that ends up a linear. image0 sdb image1 sdc log image0 sdd log image1 sde if you fail sdd and your fault policy is allocate, the mirror will end up looking like this: image0 sdb image1 sdc log sde Jon explained to me that this is expected (at least for the time being) since there is no way to go from disk to mirrored log with the current code. The original issue in this bug is fixed. The issue mentioned in comment #7 is documented in bug 607334. Marking verified in the latest rpms. 2.6.32-59.1.el6.x86_64 lvm2-2.02.72-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 lvm2-libs-2.02.72-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 lvm2-cluster-2.02.72-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 udev-147-2.22.el6 BUILT: Fri Jul 23 07:21:33 CDT 2010 device-mapper-1.02.53-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 device-mapper-libs-1.02.53-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 device-mapper-event-1.02.53-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 device-mapper-event-libs-1.02.53-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 cmirror-2.02.72-8.el6 BUILT: Wed Aug 18 10:41:52 CDT 2010 Red Hat Enterprise Linux 6.0 is now available and should resolve the problem described in this bug report. This report is therefore being closed with a resolution of CURRENTRELEASE. You may reopen this bug report if the solution does not work for you. |
Description of problem: Scenario: Kill primary log of synced 2 leg redundand log mirror(s) ********* Mirror hash info for this scenario ********* * names: syncd_primary_log_2legs_2logs_1 * sync: 1 * leg devices: /dev/sdg1 /dev/sdb1 * log devices: /dev/sdh1 /dev/sdd1 * failpv(s): /dev/sdh1 * failnode(s): taft-01 * leg fault policy: remove * log fault policy: allocate ****************************************************** Creating mirror(s) on taft-01... taft-01: lvcreate --mirrorlog mirrored -m 1 -n syncd_primary_log_2legs_2logs_1 -L 600M helter_skelter /dev/sdg1:0-1000 /dev/sdb1:0-1000 /dev/sdh1:0-150 /dev/sdd1:0-150 PV=/dev/sdh1 syncd_primary_log_2legs_2logs_1_mlog_mimage_0: 1.2 PV=/dev/sdh1 syncd_primary_log_2legs_2logs_1_mlog_mimage_0: 1.2 Waiting until all mirrors become fully syncd... 0/1 mirror(s) are fully synced: ( 63.25% ) 1/1 mirror(s) are fully synced: ( 100.00% ) Creating ext on top of mirror(s) on taft-01... mke2fs 1.41.10 (10-Feb-2009) Mounting mirrored ext filesystems on taft-01... Writing verification files (checkit) to mirror(s) on... ---- taft-01 ---- Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure Verifying files (checkit) on mirror(s) on... ---- taft-01 ---- Disabling device sdh on taft-01 Attempting I/O to cause mirror down conversion(s) on taft-01 10+0 records in 10+0 records out 41943040 bytes (42 MB) copied, 0.240536 s, 174 MB/s Verifying current sanity of lvm after the failure /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error [...] Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v. Verifying FAILED device /dev/sdh1 is *NOT* in the volume(s) /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error [...] Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v. Verifying LOG device(s) /dev/sdh1 /dev/sdd1 are *NOT* in the linear(s) /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error [...] Couldn't find device with uuid w56nnc-636G-ifll-2dH4-WJHb-ZvXC-P28E2v. log device /dev/sdd1 should no longer be present on taft-01 # mirror before the failure syncd_primary_log_2legs_2logs_1 helter_skelter mwi-ao 600.00m syncd_primary_log_2legs_2logs_1_mlog 100.00 syncd_primary_log_2legs_2logs_1_mimage_0(0),syncd_primary_log_2legs_2logs_1_mimage_1(0) [syncd_primary_log_2legs_2logs_1_mimage_0] helter_skelter iwi-ao 600.00m /dev/sdg1(0) [syncd_primary_log_2legs_2logs_1_mimage_1] helter_skelter iwi-ao 600.00m /dev/sdb1(0) [syncd_primary_log_2legs_2logs_1_mlog] helter_skelter mwi-ao 4.00m 100.00 syncd_primary_log_2legs_2logs_1_mlog_mimage_0(0),syncd_primary_log_2legs_2logs_1_mlog_mimage_1(0) [syncd_primary_log_2legs_2logs_1_mlog_mimage_0] helter_skelter iwi-ao 4.00m /dev/sdh1(0) [syncd_primary_log_2legs_2logs_1_mlog_mimage_1] helter_skelter iwi-ao 4.00m /dev/sdd1(0) # mirror after the failure syncd_primary_log_2legs_2logs_1 helter_skelter mwi-ao 600.00m syncd_primary_log_2legs_2logs_1_mlog 100.00 syncd_primary_log_2legs_2logs_1_mimage_0(0),syncd_primary_log_2legs_2logs_1_mimage_1(0) [syncd_primary_log_2legs_2logs_1_mimage_0] helter_skelter iwi-ao 600.00m /dev/sdg1(0) [syncd_primary_log_2legs_2logs_1_mimage_1] helter_skelter iwi-ao 600.00m /dev/sdb1(0) [syncd_primary_log_2legs_2logs_1_mlog] helter_skelter lwi-ao 4.00m /dev/sdd1(0) syncd_primary_log_2legs_2logs_1_mlog_mimage_0 helter_skelter -wi-a- 4.00m unknown device(0) syncd_primary_log_2legs_2logs_1_mlog_mimage_1 helter_skelter vwi-a- 4.00m # console Jun 23 18:59:02 taft-01 kernel: sd 3:0:0:7: rejecting I/O to offline device Jun 23 18:59:02 taft-01 lvm[1950]: LV helter_skelter/syncd_primary_log_2legs_2logs_1_mlog_mimage_0 in use: not deactivating Jun 23 18:59:02 taft-01 lvm[1950]: Repair of mirrored LV helter_skelter/syncd_primary_log_2legs_2logs_1 failed. Jun 23 18:59:02 taft-01 lvm[1950]: Failed to remove faulty devices in helter_skelter-syncd_primary_log_2legs_2logs_1_mlog. Jun 23 18:59:02 taft-01 kernel: sd 3:0:0:7: rejecting I/O to offline device Version-Release number of selected component (if applicable): Version 2.02.67+ (custom build)