Bug 1674549

Summary:	[cee/sd][ceph-mgr] luminous: deadlock in standby ceph-mgr daemons
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Tomas Petr <tpetr>
Component:	RADOS	Assignee:	Brad Hubbard <bhubbard>
Status:	CLOSED ERRATA	QA Contact:	Manohar Murthy <mmurthy>
Severity:	medium	Docs Contact:	Aron Gunn <agunn>
Priority:	medium
Version:	3.2	CC:	agunn, anharris, assingh, bhubbard, branto, ceph-eng-bugs, dzafman, gsitlani, jbrier, kchai, mhackett, nojha, tchandra, tserlin, vumrao
Target Milestone:	z2
Target Release:	3.2
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	RHEL: ceph-12.2.8-113.el7cp Ubuntu: ceph_12.2.8-96redhat1xenial	Doc Type:	Bug Fix
Doc Text:	.A race condition was causing threads to deadlock with the standby `ceph-mgr` daemon Some threads can cause a race condition when acquiring a local lock and the Python global interpreter lock, which is causing a deadlock issue for each thread. As the thread holds on to one of the locks, it wants to acquire the other lock, but cannot. In this release, the code was fixed to close the window of opportunity for the race condition to occur. This is done by changing the location of the lock acquisition and releasing the appropriate locks. Doing this results in the threads not causing a deadlock, which allows progress to be made.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-04-30 15:56:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1629656

Description Tomas Petr 2019-02-11 15:40:22 UTC

Description of problem:
From upstream tracker:
http://tracker.ceph.com/issues/35985
StandbyPyModule::get_config is using state.with_config without dropping the GIL around taking the lock.


The standby mgr process hangs without response, it is removed from mgrmap and does not retake active role when active mgr stops.

without MGR daemon, ceph reports 0 space, which has impact on OSP spawning new instances, as the available space is checked.

Version-Release number of selected component (if applicable):
12.2.8-52

How reproducible:
Random

Steps to Reproduce:
1.
2.
3.

Actual results:
standby mgr process stops responding, but process is still running - no msgs logged

Expected results:
standby mgr process is responding, if active mgr stops, one of stanbys mgr become active

Additional info:

Comment 5 Brad Hubbard 2019-02-22 23:54:04 UTC

See analysis in  https://tracker.ceph.com/issues/35985

Comment 6 Brad Hubbard 2019-02-24 21:42:45 UTC

https://github.com/ceph/ceph/pull/26613

Comment 10 Brad Hubbard 2019-03-14 22:03:13 UTC

We are still waiting on thread dumps to confirm this issue is the same as https://tracker.ceph.com/issues/35985.

Comment 19 errata-xmlrpc 2019-04-30 15:56:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:0911