Bug 678645

Summary: dmeventd died during mirror w/ snapshot device failure
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Petr Rockai <prockai>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Corey Marthaler <cmarthal>
Severity: high Docs Contact:
Priority: high    
Version: 6.1CC: agk, dwysocha, heinzm, jbrassow, mbroz, prajnoha, prockai, thornber, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-03-22 14:59:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
log from taft-01
none
original core dump from taft-01 none

Description Corey Marthaler 2011-02-18 18:06:10 UTC 
Description of problem:
This is for the issue listed in comment #27 of bug 613829. It's currently only been seen once.

Scenario: Kill disk log of synced 2 leg mirror(s)

********* Mirror hash info for this scenario *********
* names:              syncd_log_2legs_1
* sync:               1
* leg devices:        /dev/sdd1 /dev/sdg1
* log devices:        /dev/sdf1
* no MDA devices:     
* failpv(s):          /dev/sdf1
* failnode(s):        taft-01
* additional snap:    /dev/sdd1
* leg fault policy:   remove
* log fault policy:   allocate
******************************************************

Creating mirror(s) on taft-01...
taft-01: lvcreate -m 1 -n syncd_log_2legs_1 -L 600M helter_skelter
/dev/sdd1:0-1000 /dev/sdg1:0-1000 /dev/sdf1:0-150
Creating a snapshot volume of each of the mirrors

Waiting until all mirrors become fully syncd...
   0/1 mirror(s) are fully synced: ( 96.83% )
   1/1 mirror(s) are fully synced: ( 100.00% )

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----

Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Disabling device sdf on taft-01
[DEADLOCK]

[root@taft-01 ~]# lvs -a -o +devices
[DEADLOCK]


Looks like dmevetd died during this:
Feb  9 16:56:58 taft-01 abrt[6704]: saved core dump of pid 1222
(/sbin/dmeventd) to /var/spool/abrt/ccpp-1297292218-1222.new/coredump (32448512
bytes)
Feb  9 16:56:58 taft-01 abrtd: Directory 'ccpp-1297292218-1222' creation
detected
Feb  9 16:56:58 taft-01 abrtd: Registered Database plugin 'SQLite3'
Feb  9 16:56:58 taft-01 abrtd: New crash /var/spool/abrt/ccpp-1297292218-1222,
processing

2.6.32-94.el6.x86_64

lvm2-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
lvm2-libs-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
lvm2-cluster-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
udev-147-2.31.el6    BUILT: Wed Jan 26 05:39:15 CST 2011
device-mapper-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
device-mapper-libs-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
device-mapper-event-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
device-mapper-event-libs-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
cmirror-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
Comment 1 Corey Marthaler 2011-02-18 18:07:23 UTC 
The coredump is too big to attach here. So I'll keep it on
taft-01:/var/spool/abrt/ccpp-1297292218-1222 and in my home dir
/home/msp/cmarthal/678645.
Comment 2 Corey Marthaler 2011-02-18 18:11:33 UTC 
Created attachment 479582 [details]
log from taft-01
Comment 3 RHEL Program Management 2011-04-04 02:03:52 UTC 
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 5 Petr Rockai 2011-06-02 21:17:23 UTC 
Of course the core dump no longer exists on the machine. Meaning this is probably not going to be tracked down unless it can be reproduced. Corey, any chance on repeating the crash? (Or, alternatively, any chance of the core dump being stashed somewhere still, ideally with a matching dmeventd binary? I don't have access to your $HOME, so it may still be there.)
Comment 6 Corey Marthaler 2011-06-02 21:28:03 UTC 
Created attachment 502653 [details]
original core dump from taft-01
Comment 7 RHEL Program Management 2011-10-07 15:55:01 UTC 
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 8 Milan Broz 2012-03-22 14:59:27 UTC 
Closing this, many changes in dmeventd since then, please reopen if you see coredump again with new builds. thanks.