Bug 429708
| Summary: | RHEL5 cmirror tracker: cluster locking deadlock | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Corey Marthaler <cmarthal> | ||||||||||||||
| Component: | cmirror | Assignee: | Jonathan Earl Brassow <jbrassow> | ||||||||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Cluster QE <mspqa-list> | ||||||||||||||
| Severity: | low | Docs Contact: | |||||||||||||||
| Priority: | low | ||||||||||||||||
| Version: | 5.2 | CC: | agk, ccaulfie, dwysocha, heinzm, mbroz | ||||||||||||||
| Target Milestone: | rc | Keywords: | TestBlocker | ||||||||||||||
| Target Release: | --- | ||||||||||||||||
| Hardware: | All | ||||||||||||||||
| OS: | Linux | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||
| Last Closed: | 2008-10-15 21:56:05 UTC | Type: | --- | ||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
| Embargoed: | |||||||||||||||||
| Bug Depends On: | |||||||||||||||||
| Bug Blocks: | 430797 | ||||||||||||||||
| Attachments: |
|
||||||||||||||||
|
Description
Corey Marthaler
2008-01-22 15:59:04 UTC
Created attachment 292527 [details]
backtraces from hayes-01
Created attachment 292528 [details]
backtraces from hayes-02
Created attachment 292529 [details]
backtraces from hayes-03
What are "cmirror config operations"? converting/creating/removing? I haven't looked at the attachments yet, but it doesn't look like it's hit cmirror code yet... LVM/device-mapper has changed quite a bit in this release - it could be there too. Something that convinces you that it's cmirror? need -vvvv for all three commands to check for new userspace deadlocks - or lock manager state showing which locks are held where in the cluster The operations, run from multiple nodes in the cluster to mulitple mirrors, are as follows: create, down convert, up convert, core log convert, disk log convert, deactivate, reactivate, deactivate, delete. This deadlock can be triggered by just having many mirrors to activate during clvmd start up. many == ? There were 28 mirrors in that case, however I spoke to soon, it didn't deadlock, it everntually timed out and then the stream of 'dm-log-clustered:' errors followed, so comment #8 is most likely one of the other open BZs corey, is this still reproducible? I'm unable to reproducing this, so this may be fixed. However I do eventually hit another bz so verifying this may be blocked behind 429599. I think 429599 has been cleared... ok to have another go at this? I appear to have hit this last night, again while running cirror_lock_stress. ./cmirror_lock_stress -l /home/msp/cmarthal/work/rhel5/sts-root -r /usr/tests/sts-rhel5.2 -R ../../var/share/resource_files/hayes.xml -m 2 I saw a lot of the following messages on the console: "dm-log-clustered: Stray request returned:" "dm-log-clustered: Excessive delay in request processing, 1 sec" There are no mirror or mirror components currently listed with dmsetup I'll attach a dump of the stuck processes... [root@hayes-01 ~]# rpm -qi lvm2 Name : lvm2 Relocations: (not relocatable) Version : 2.02.32 Vendor: Red Hat, Inc. Release : 4.el5 Build Date: Fri 04 Apr 2008 06:15:19 AM CDT Install Date: Mon 05 May 2008 11:26:24 AM CDT Build Host: hs20-bc2-3.build.redhat.com [root@hayes-01 ~]# rpm -qi lvm2-cluster Name : lvm2-cluster Relocations: (not relocatable) Version : 2.02.32 Vendor: Red Hat, Inc. Release : 4.el5 Build Date: Wed 02 Apr 2008 03:56:50 AM CDT Install Date: Mon 05 May 2008 11:27:46 AM CDT Build Host: hs20-bc2-3.build.redhat.com [root@hayes-01 ~]# rpm -qi cmirror Name : cmirror Relocations: (not relocatable) Version : 1.1.17 Vendor: Red Hat, Inc. Release : 1.el5 Build Date: Fri 09 May 2008 11:33:43 AM CDT Install Date: Fri 09 May 2008 03:57:23 PM CDT Build Host: hs20-bc1-7.build.redhat.com [root@hayes-01 ~]# rpm -qi kmod-cmirror Name : kmod-cmirror Relocations: (not relocatable) Version : 0.1.9 Vendor: Red Hat, Inc. Release : 1.el5 Build Date: Thu 08 May 2008 02:28:27 PM CDT Install Date: Fri 09 May 2008 09:17:33 AM CDT Build Host: hs20-bc2-4.build.redhat.com [root@hayes-01 ~]# rpm -qi openais Name : openais Relocations: (not relocatable) Version : 0.80.3 Vendor: Red Hat, Inc. Release : 15.el5 Build Date: Wed 02 Apr 2008 03:42:29 AM CDT Install Date: Mon 05 May 2008 11:26:27 AM CDT Build Host: ls20-bc2-13.build.redhat.com Created attachment 305252 [details]
log from hayes-01
Created attachment 305253 [details]
log from hayes-02
Created attachment 305254 [details]
log from hayes-03
This bz has not been seen in 5 months, or is possibly the same as bz 460222. Will reopen if needed. |