Bug 246630

Summary:	clvmd segfault when view of devices across cluster is inconsistent
Product:	[Retired] Red Hat Cluster Suite	Reporter:	Corey Marthaler <cmarthal>
Component:	lvm2-cluster	Assignee:	LVM and device-mapper development team <lvm-team>
Status:	CLOSED WONTFIX	QA Contact:	Cluster QE <mspqa-list>
Severity:	low	Docs Contact:
Priority:	low
Version:	4	CC:	agk, ccaulfie, dwysocha, jbrassow, mbroz, prockai
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-05-07 20:48:00 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Corey Marthaler 2007-07-03 14:51:56 UTC

Description of problem:
This may be related to bz 230437 or 236432. I had I/O going to 2 cmirrors (both
on the same vg) and then failed the primary leg of the first mirror and the log
of the second mirror. I staggered the failures so that they didn't happen right
away on all the nodes, first the failure happened on link-02 and link-08, and
then on link-04 a few seconds afterwards. Link-04 was the node where clvmd died.


Jul  2 15:17:25 link-04 kernel: clvmd[4622]: segfault at 0000000000000040 rip
0000003694470240 rsp 0000000041400a88 error 4


Version-Release number of selected component (if applicable):
2.6.9-55.8.ELsmp
lvm2-cluster-2.02.21-7.el4
cmirror-kernel-2.6.9-32.0

Comment 1 Corey Marthaler 2007-07-03 15:02:59 UTC

More info...


[root@link-02 ~]# dmsetup ls
corey-mirror1_mlog      (253, 4)
corey-mirror2_mlog      (253, 9)
corey-mirror2   (253, 13)
corey-mirror2_mimage_2  (253, 12)
vg3-lv  (253, 2)
corey-mirror2_mimage_1  (253, 11)
corey-mirror1   (253, 8)
corey-mirror1_mimage_2  (253, 7)
corey-mirror2_mimage_0  (253, 10)
corey-mirror1_mimage_1  (253, 6)
corey-mirror1_mimage_0  (253, 5)
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
vg3-lv2 (253, 3)
[root@link-02 ~]# dmsetup info
Name:              corey-mirror1_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDxQFY33GCnoJAyyyrD8h517k7PlBMXIRc

Name:              corey-mirror2_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 9
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDBGgkXEwHwEgvXTNQDaZkdGyVVfcKS41Y

Name:              corey-mirror2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      2
Major, minor:      253, 13
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDozfB3UJ7xoShAIjBjYBkR03H4Olw6yKA

Name:              corey-mirror2_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 12
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDOtaBAo3Ar20SMoLaUfkhVxzq0yPbaPid

Name:              vg3-lv
State:             ACTIVE
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 2
Number of targets: 2
UUID: LVM-3VUB1rHY0QNPWfH6YrwZhUUTxKo41qU5cO1iSvzvrDDZx1BCcB8I3qxDx4AOkRgl

Name:              corey-mirror2_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 11
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolD4I4eSG94W7Zg1zDjOStTRd5689uWuS02

Name:              corey-mirror1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      37
Major, minor:      253, 8
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDR2gVZq2On1lhhFf4XmxdZdlez9tQqqPo

Name:              corey-mirror1_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 7
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDJgtyxSwuLBoZIkltI8dzr2XmIipDOvpG

Name:              corey-mirror2_mimage_0
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 10
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolD979jr553M3muiWfitn1P0P7WI59d7qaq

Name:              corey-mirror1_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 6
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDOuUnhqCOV2yGOzScJiUZAB1BqPgX6sQf

Name:              corey-mirror1_mimage_0
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDfH0E75Z3x0kVKxhstOg05kFxz1qC3AtO

Name:              VolGroup00-LogVol01
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-dq1liKVsB8CzZiuNRtOF1tYTkXqgKO85b28r2eyrPqurvpuPPS83R8fcaG7QULHG

Name:              VolGroup00-LogVol00
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-dq1liKVsB8CzZiuNRtOF1tYTkXqgKO85W9BzgAV2x8i4cT3pKw0TArv1HJm54Tu6

Name:              vg3-lv2
State:             ACTIVE
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-3VUB1rHY0QNPWfH6YrwZhUUTxKo41qU5PJ3zFIvjhZh6gXg7G6aNZCGMmkkqseg7



[root@link-04 ~]# dmsetup ls
corey-mirror1_mlog      (253, 2)
corey-mirror2_mlog      (253, 7)
corey-mirror2   (253, 11)
corey-mirror2_mimage_2  (253, 10)
corey-mirror2_mimage_1  (253, 9)
corey-mirror1   (253, 6)
corey-mirror1_mimage_2  (253, 5)
corey-mirror2_mimage_0  (253, 8)
corey-mirror1_mimage_1  (253, 4)
corey-mirror1_mimage_0  (253, 3)
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
[root@link-04 ~]# dmsetup info
Name:              corey-mirror1_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDxQFY33GCnoJAyyyrD8h517k7PlBMXIRc

Name:              corey-mirror2_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 7
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDBGgkXEwHwEgvXTNQDaZkdGyVVfcKS41Y

Name:              corey-mirror2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      2
Major, minor:      253, 11
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDozfB3UJ7xoShAIjBjYBkR03H4Olw6yKA

Name:              corey-mirror2_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 10
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDOtaBAo3Ar20SMoLaUfkhVxzq0yPbaPid

Name:              corey-mirror2_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 9
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolD4I4eSG94W7Zg1zDjOStTRd5689uWuS02

Name:              corey-mirror1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      5
Major, minor:      253, 6
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDR2gVZq2On1lhhFf4XmxdZdlez9tQqqPo

Name:              corey-mirror1_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDJgtyxSwuLBoZIkltI8dzr2XmIipDOvpG

Name:              corey-mirror2_mimage_0
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 8
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolD979jr553M3muiWfitn1P0P7WI59d7qaq

Name:              corey-mirror1_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDOuUnhqCOV2yGOzScJiUZAB1BqPgX6sQf

Name:              corey-mirror1_mimage_0
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDfH0E75Z3x0kVKxhstOg05kFxz1qC3AtO

Name:              VolGroup00-LogVol01
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-RsBOCkblQpLicArLlocNswlochRxjsFjyYdc3o77yCGdTIILMdW0LlXkWPGcQWen

Name:              VolGroup00-LogVol00
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-RsBOCkblQpLicArLlocNswlochRxjsFjRMSyxR0nBvsyde1ijAWS9nf4aX3gDTYI



[root@link-08 ~]# dmsetup ls
corey-mirror1_mlog      (253, 4)
corey-mirror2_mlog      (253, 9)
corey-mirror2   (253, 13)
corey-mirror2_mimage_2  (253, 12)
vg3-lv  (253, 2)
corey-mirror2_mimage_1  (253, 11)
corey-mirror1   (253, 8)
corey-mirror1_mimage_2  (253, 7)
corey-mirror2_mimage_0  (253, 10)
corey-mirror1_mimage_1  (253, 6)
corey-mirror1_mimage_0  (253, 5)
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
vg3-lv2 (253, 3)
[root@link-08 ~]# dmsetup info
Name:              corey-mirror1_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDxQFY33GCnoJAyyyrD8h517k7PlBMXIRc

Name:              corey-mirror2_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 9
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDBGgkXEwHwEgvXTNQDaZkdGyVVfcKS41Y

Name:              corey-mirror2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      7
Major, minor:      253, 13
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDozfB3UJ7xoShAIjBjYBkR03H4Olw6yKA

Name:              corey-mirror2_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 12
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDOtaBAo3Ar20SMoLaUfkhVxzq0yPbaPid

Name:              vg3-lv
State:             ACTIVE
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 2
Number of targets: 2
UUID: LVM-3VUB1rHY0QNPWfH6YrwZhUUTxKo41qU5cO1iSvzvrDDZx1BCcB8I3qxDx4AOkRgl

Name:              corey-mirror2_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 11
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolD4I4eSG94W7Zg1zDjOStTRd5689uWuS02

Name:              corey-mirror1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      39
Major, minor:      253, 8
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDR2gVZq2On1lhhFf4XmxdZdlez9tQqqPo

Name:              corey-mirror1_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 7
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDJgtyxSwuLBoZIkltI8dzr2XmIipDOvpG

Name:              corey-mirror2_mimage_0
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 10
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolD979jr553M3muiWfitn1P0P7WI59d7qaq

Name:              corey-mirror1_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 6
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDOuUnhqCOV2yGOzScJiUZAB1BqPgX6sQf

Name:              corey-mirror1_mimage_0
State:             SUSPENDED
Tables present:    LIVE & INACTIVE
Open count:        2
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: LVM-Ob03M1RdEgA7ZokFp1YfBD6RSBJpRolDfH0E75Z3x0kVKxhstOg05kFxz1qC3AtO

Name:              VolGroup00-LogVol01
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-TCx5xJ7FuRhXzJ4g7CvPsw2AhhFBNLQUvNlDc7SvClgdBMh2WD6TraPFgzjSVMRp

Name:              VolGroup00-LogVol00
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-TCx5xJ7FuRhXzJ4g7CvPsw2AhhFBNLQUrfFjpdEgWnBeJx2UpyC5Mr6XpgXdHWCh

Name:              vg3-lv2
State:             ACTIVE
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-3VUB1rHY0QNPWfH6YrwZhUUTxKo41qU5PJ3zFIvjhZh6gXg7G6aNZCGMmkkqseg7

Comment 2 Corey Marthaler 2007-07-03 15:06:07 UTC

more info...

[root@link-02 ~]# cat /proc/cluster/nodes
Node  Votes Exp Sts  Name
   1    1    3   M   link-02
   2    1    3   M   link-04
   3    1    3   M   link-08
[root@link-02 ~]# cat /proc/cluster/services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           4   2 run       -
[1 3 2]

DLM Lock Space:  "clvmd"                             5   3 run       -
[1 3 2]

DLM Lock Space:  "clustered_log"                     6   4 run       -
[1 3 2]

DLM Lock Space:  "1"                                 7   5 run       -
[1 3 2]

DLM Lock Space:  "2"                                 9   7 run       -
[1 3 2]

GFS Mount Group: "1"                                 8   6 run       -
[1 3 2]

GFS Mount Group: "2"                                10   8 run       -
[1 3 2]



[root@link-04 ~]# cat /proc/cluster/nodes
Node  Votes Exp Sts  Name
   1    1    3   M   link-02
   2    1    3   M   link-04
   3    1    3   M   link-08
[root@link-04 ~]# cat /proc/cluster/services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           4   2 run       -
[1 3 2]

DLM Lock Space:  "clvmd"                             5   3 run       -
[1 3 2]

DLM Lock Space:  "clustered_log"                     6   4 run       -
[1 3 2]

DLM Lock Space:  "1"                                 7   5 run       -
[1 3 2]

DLM Lock Space:  "2"                                 9   7 run       -
[1 3 2]

GFS Mount Group: "1"                                 8   6 run       -
[1 3 2]

GFS Mount Group: "2"                                10   8 run       -
[1 3 2]


[root@link-08 ~]# cat /proc/cluster/nodes
Node  Votes Exp Sts  Name
   1    1    3   M   link-02
   2    1    3   M   link-04
   3    1    3   M   link-08
[root@link-08 ~]# cat /proc/cluster/services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           4   2 run       -
[3 1 2]

DLM Lock Space:  "clvmd"                             5   3 run       -
[1 3 2]

DLM Lock Space:  "clustered_log"                     6   4 run       -
[1 3 2]

DLM Lock Space:  "1"                                 7   5 run       -
[1 3 2]

DLM Lock Space:  "2"                                 9   7 run       -
[1 3 2]

GFS Mount Group: "1"                                 8   6 run       -
[1 3 2]

GFS Mount Group: "2"                                10   8 run       -
[1 3 2]

Comment 3 Corey Marthaler 2007-07-03 15:16:50 UTC

[root@link-02 tmp]# ./lvm_backtraces.pl
Jul  3 10:07:29 link-02 kernel: dm-cmirror: LOG INFO:
Jul  3 10:07:29 link-02 kernel: dm-cmirror:   uuid: LVM-Ob03M1RdEgA7ZokFp1YfBD6R
SBJpRolDxQFY33GCnoJAyyyrD8h517k7PlBMXIRc
Jul  3 10:07:29 link-02 kernel: dm-cmirror:   uuid_ref    : 1
Jul  3 10:07:29 link-02 kernel: dm-cmirror:  ?region_count: 10240
Jul  3 10:07:29 link-02 kernel: dm-cmirror:  ?sync_count  : 10240
Jul  3 10:07:29 link-02 kernel: dm-cmirror:  ?sync_search : 5333
Jul  3 10:07:29 link-02 kernel: dm-cmirror:   in_sync     : YES
Jul  3 10:07:29 link-02 kernel: dm-cmirror:   suspended   : NO
Jul  3 10:07:29 link-02 kernel: dm-cmirror:   server_id   : [[link-02]]
Jul  3 10:07:29 link-02 kernel: dm-cmirror:   server_valid: YES
DLM lockspace 'clvmd'

Resource 00000100372d0048 (parent 0000000000000000). Name (len=64) "Ob03M1RdEgA7
ZokFp1YfBD6RSBJpRolDozfB3UJ7xoShAIjBjYBkR03H4Olw6yKA"
Local Copy, Master is node [[link-08]]
Granted Queue
00010059 CR Master:     00010107
Conversion Queue
Waiting Queue

Resource 00000100372d0e88 (parent 0000000000000000). Name (len=64) "Ob03M1RdEgA7
                                         
ZokFp1YfBD6RSBJpRolDR2gVZq2On1lhhFf4XmxdZdlez9tQqqPo"
Local Copy, Master is node [[link-08]]
Granted Queue
000101e4 CR Master:     00010318
Conversion Queue
Waiting Queue



[root@link-04 tmp]# ./lvm_backtraces.pl
Jul  3 07:40:17 link-04 dhclient: DHCPREQUEST on eth0 to 10.15.89.100 port 67
Jul  3 07:40:18 link-04 dhclient: DHCPACK from 10.15.89.100
Jul  3 07:40:18 link-04 dhclient: bound to 10.15.89.154 -- renewal in 8606 secon
ds.
Jul  3 10:01:23 link-04 sshd(pam_unix)[24356]: authentication failure; logname=
uid=0 euid=0 tty=ssh ruser= rhost=silver.msp.redhat.com  user=root
Jul  3 10:01:27 link-04 sshd(pam_unix)[24363]: session opened for user root by r
oot(uid=0)
Jul  3 10:03:43 link-04 dhclient: DHCPREQUEST on eth0 to 10.15.89.100 port 67
Jul  3 10:03:44 link-04 dhclient: DHCPACK from 10.15.89.100
Jul  3 10:03:44 link-04 dhclient: bound to 10.15.89.154 -- renewal in 9203 secon
ds.
Jul  3 10:03:59 link-04 sshd(pam_unix)[24590]: session opened for user root by r
oot(uid=0)
Jul  3 10:07:02 link-04 sshd(pam_unix)[24814]: session opened for user root by (
uid=0)
Jul  3 10:08:10 link-04 kernel: dm-cmirror: LOG INFO:
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   uuid: LVM-Ob03M1RdEgA7ZokFp1YfBD6R
SBJpRolDBGgkXEwHwEgvXTNQDaZkdGyVVfcKS41Y
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   uuid_ref    : 1
Jul  3 10:08:10 link-04 kernel: dm-cmirror:  ?region_count: 10240
Jul  3 10:08:10 link-04 kernel: dm-cmirror:  ?sync_count  : 0
Jul  3 10:08:10 link-04 kernel: dm-cmirror:  ?sync_search : 0
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   in_sync     : YES
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   suspended   : NO
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   server_id   : [[link-02]]
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   server_valid: YES
Jul  3 10:08:10 link-04 kernel: dm-cmirror: LOG INFO:
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   uuid: LVM-Ob03M1RdEgA7ZokFp1YfBD6R
SBJpRolDxQFY33GCnoJAyyyrD8h517k7PlBMXIRc
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   uuid_ref    : 1
Jul  3 10:08:10 link-04 kernel: dm-cmirror:  ?region_count: 10240
Jul  3 10:08:10 link-04 kernel: dm-cmirror:  ?sync_count  : 0
Jul  3 10:08:10 link-04 kernel: dm-cmirror:  ?sync_search : 0
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   in_sync     : YES
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   suspended   : NO
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   server_id   : [[link-02]]
Jul  3 10:08:10 link-04 kernel: dm-cmirror:   server_valid: YES
DLM lockspace 'clvmd'

Resource 000001003814b2a8 (parent 0000000000000000). Name (len=64) "Ob03M1RdEgA7
                                         
ZokFp1YfBD6RSBJpRolDfH0E75Z3x0kVKxhstOg05kFxz1qC3AtO"
Master Copy
Granted Queue
00010123 PW Remote:   [[link-08]] 0001017a
Conversion Queue
Waiting Queue



[root@link-08 tmp]# ./lvm_backtraces.pl
Backtrace for lvs-a-o+devices (9157):
corey-mirror1_mimage_0 is SUSPENDED
Jul  3 03:39:58 link-08 dhclient: bound to 10.15.89.158 -- renewal in 9770 secon
ds.
Jul  3 06:22:48 link-08 dhclient: DHCPREQUEST on eth0 to 10.15.89.100 port 67
Jul  3 06:22:48 link-08 dhclient: DHCPACK from 10.15.89.100
Jul  3 06:22:48 link-08 dhclient: bound to 10.15.89.158 -- renewal in 8670 secon
ds.
Jul  3 08:47:18 link-08 dhclient: DHCPREQUEST on eth0 to 10.15.89.100 port 67
Jul  3 08:47:18 link-08 dhclient: DHCPACK from 10.15.89.100
Jul  3 08:47:18 link-08 dhclient: bound to 10.15.89.158 -- renewal in 8135 secon
ds.
Jul  3 10:08:38 link-08 sshd(pam_unix)[11903]: session opened for user root by r
oot(uid=0)
Jul  3 10:10:46 link-08 sshd(pam_unix)[12069]: session opened for user root by r
oot(uid=0)
Jul  3 10:13:52 link-08 sshd(pam_unix)[12288]: session opened for user root by (
uid=0)
Jul  3 10:14:56 link-08 kernel: dm-cmirror: LOG INFO:
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   uuid: LVM-Ob03M1RdEgA7ZokFp1YfBD6R
SBJpRolDBGgkXEwHwEgvXTNQDaZkdGyVVfcKS41Y
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   uuid_ref    : 1
Jul  3 10:14:56 link-08 kernel: dm-cmirror:  ?region_count: 10240
Jul  3 10:14:56 link-08 kernel: dm-cmirror:  ?sync_count  : 0
Jul  3 10:14:56 link-08 kernel: dm-cmirror:  ?sync_search : 0
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   in_sync     : YES
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   suspended   : NO
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   server_id   : [[link-02]]
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   server_valid: YES
Jul  3 10:14:56 link-08 kernel: dm-cmirror: LOG INFO:
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   uuid: LVM-Ob03M1RdEgA7ZokFp1YfBD6R
SBJpRolDxQFY33GCnoJAyyyrD8h517k7PlBMXIRc
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   uuid_ref    : 1
Jul  3 10:14:56 link-08 kernel: dm-cmirror:  ?region_count: 10240
Jul  3 10:14:56 link-08 kernel: dm-cmirror:  ?sync_count  : 0
Jul  3 10:14:56 link-08 kernel: dm-cmirror:  ?sync_search : 0
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   in_sync     : YES
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   suspended   : NO
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   server_id   : [[link-02]]
Jul  3 10:14:56 link-08 kernel: dm-cmirror:   server_valid: YES
DLM lockspace 'clvmd'

Resource 000001003c1bf048 (parent 0000000000000000). Name (len=64) "Ob03M1RdEgA7
ZokFp1YfBD6RSBJpRolDozfB3UJ7xoShAIjBjYBkR03H4Olw6yKA"
Master Copy
Granted Queue
00010107 CR Remote:   [[link-02]] 00010059
000100b2 CR
Conversion Queue
Waiting Queue

Resource 000001002cbcfc78 (parent 0000000000000000). Name (len=7) "V_corey"
Master Copy
Granted Queue
000103bc PR
Conversion Queue
Waiting Queue

Resource 000001003c1bfd58 (parent 0000000000000000). Name (len=64) "Ob03M1RdEgA7
                                         
ZokFp1YfBD6RSBJpRolDR2gVZq2On1lhhFf4XmxdZdlez9tQqqPo"
Master Copy
Granted Queue
00010318 CR Remote:   [[link-02]] 000101e4
00010390 CR
Conversion Queue
Waiting Queue

Resource 000001003c1bfe88 (parent 0000000000000000). Name (len=64) "Ob03M1RdEgA7
                                         
ZokFp1YfBD6RSBJpRolDfH0E75Z3x0kVKxhstOg05kFxz1qC3AtO"
Local Copy, Master is node [[link-04]]
Granted Queue
0001017a PW Master:     00010123
Conversion Queue
Waiting Queue

Comment 4 Corey Marthaler 2007-07-03 16:44:02 UTC

This bug appears to be reproducable. I was able to hit the deadlock again after
failing the leg and log like in comment #0, however there was no clvmd segfault
this time.

Comment 5 Jonathan Earl Brassow 2007-09-28 15:33:58 UTC

So, you are killing devices, but at different times on the various machines?

This would produce a problem where clvmd sees and inconsistent view of
devices... bug 249092 also falls into this catagory.   We may wish to round up
all the bugs that fall into this catagory and create a bug that references them.

This is a known limitation of clvmd capabilities and should be addressed.

Comment 6 Jonathan Earl Brassow 2008-03-26 18:19:40 UTC

Changing subject to more accurately describe problem

Comment 7 Jonathan Earl Brassow 2008-03-26 18:30:32 UTC

*** Bug 230437 has been marked as a duplicate of this bug. ***

Comment 9 Jonathan Earl Brassow 2010-05-07 20:48:00 UTC

CLVM does not and has never supported a device failure for just a subset of machines in a cluster.  This should be viewed as a limitation of CLVM.  If we are going to address it, it will be too intrusive for RHEL4.