Bug 619218 - subset cmirror device failure doesn't work due to locking issues
subset cmirror device failure doesn't work due to locking issues
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.0
All Linux
high Severity high
: rc
: ---
Assigned To: Jonathan Earl Brassow
Corey Marthaler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-28 17:52 EDT by Corey Marthaler
Modified: 2010-11-12 16:35 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-12 16:35:00 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2010-07-28 17:52:05 EDT
Description of problem:
Scenario: Kill secondary leg of non synced 3 leg mirror(s)

********* Mirror hash info for this scenario *********
* names:              nonsyncd_secondary_3legs_1
* sync:               0
* leg devices:        /dev/sdf1 /dev/sdb1 /dev/sdg1
* log devices:        /dev/sde1
* failpv(s):          /dev/sdb1
* failnode(s):        taft-01 taft-02
* leg fault policy:   remove
* log fault policy:   allocate
******************************************************

Creating mirror(s) on taft-04...
taft-04: lvcreate -m 2 -n nonsyncd_secondary_3legs_1 -L 600M helter_skelter /dev/sdf1:0-1000 /dev/sdb1:0-1000 /dev/sdg1:0-1000 /dev/sde1:0-150

PV=/dev/sdb1
        nonsyncd_secondary_3legs_1_mimage_1: 6
PV=/dev/sdb1
        nonsyncd_secondary_3legs_1_mimage_1: 6

Continuing on without fully syncd mirrors, currently at...
        ( 9.75% )

Creating gfs2 on top of mirror(s) on taft-01...
Mounting mirrored gfs2 filesystems on taft-01...
Mounting mirrored gfs2 filesystems on taft-02...
Mounting mirrored gfs2 filesystems on taft-03...
Mounting mirrored gfs2 filesystems on taft-04...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----
        ---- taft-02 ----
        ---- taft-03 ----
        ---- taft-04 ----

Sleeping 10 seconds to get some outsanding GFS I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----
        ---- taft-02 ----
        ---- taft-03 ----
        ---- taft-04 ----

Disabling device sdb on taft-01
Disabling device sdb on taft-02

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.311652 s, 135 MB/s
[HANG]

[root@taft-02 ~]# lvs -a -o +devices
[HANG]

LOG:
Jul 28 21:33:41 taft-01 kernel: sd 3:0:0:1: rejecting I/O to offline device
Jul 28 21:33:41 taft-01 kernel: sd 3:0:0:1: rejecting I/O to offline device
Jul 28 21:33:41 taft-01 lvm[3270]: Error locking on node taft-04: Volume group for uuid not found: gvrLI5drgY01r0NCH8kcAZ2V85tEKN1Zb8rX5MNBgXVps8qFyg1KBB53PVI0ODLn
Jul 28 21:33:41 taft-01 lvm[3270]: Error locking on node taft-03: Volume group for uuid not found: gvrLI5drgY01r0NCH8kcAZ2V85tEKN1Zb8rX5MNBgXVps8qFyg1KBB53PVI0ODLn
Jul 28 21:33:41 taft-01 lvm[3270]: Failed to lock nonsyncd_secondary_3legs_1
Jul 28 21:33:41 taft-01 lvm[3270]: Repair of mirrored LV helter_skelter/nonsyncd_secondary_3legs_1 failed.
Jul 28 21:33:41 taft-01 lvm[3270]: Failed to remove faulty devices in helter_skelter-nonsyncd_secondary_3legs_1.
Jul 28 21:33:43 taft-01 lvm[3270]: No longer monitoring mirror device helter_skelter-nonsyncd_secondary_3legs_1 for events.
Jul 28 21:36:41 taft-01 kernel: INFO: task flush-253:7:3276 blocked for more than 120 seconds.
Jul 28 21:36:41 taft-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 28 21:36:41 taft-01 kernel: flush-253:7   D ffff88021fc24100     0  3276      2 0x00000080
Jul 28 21:36:41 taft-01 kernel: ffff88021453ba20 0000000000000046 0000000000000000 ffffffff8110cd47
Jul 28 21:36:41 taft-01 kernel: ffff880200502268 ffffea0006d8c3a8 ffff88021453ba10 00000001001f20f7
Jul 28 21:36:41 taft-01 kernel: ffff880214ca7a98 ffff88021453bfd8 0000000000010518 ffff880214ca7a98
Jul 28 21:36:41 taft-01 kernel: Call Trace:
Jul 28 21:36:41 taft-01 kernel: [<ffffffff8110cd47>] ? unlock_page+0x27/0x30
Jul 28 21:36:41 taft-01 kernel: [<ffffffffa04291f0>] ? gfs2_get_block_noalloc+0x0/0x40 [gfs2]
Jul 28 21:36:41 taft-01 kernel: [<ffffffffa041f0c0>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]

Version-Release number of selected component (if applicable):
2.6.32-52.el6.x86_64

lvm2-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
lvm2-libs-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
lvm2-cluster-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
udev-147-2.21.el6    BUILT: Mon Jul 12 04:55:00 CDT 2010
device-mapper-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
device-mapper-libs-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
device-mapper-event-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
device-mapper-event-libs-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
cmirror-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
Comment 3 RHEL Product and Program Management 2010-11-12 16:35:00 EST
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.

Note You need to log in before you can comment on or make changes to this bug.