Bug 1727068

Summary:	Deadlock when generating statedumps
Product:	[Community] GlusterFS	Reporter:	Xavi Hernandez <jahernan>
Component:	core	Assignee:	Xavi Hernandez <jahernan>
Status:	CLOSED NEXTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	mainline	CC:	bugs, pasik
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1729950 1729952 (view as bug list)		Environment:
Last Closed:	2019-07-15 06:24:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1722209, 1729950, 1729952

Description Xavi Hernandez 2019-07-04 11:12:29 UTC

Description of problem:

There seems to be a deadlock between statedump generation and creation of an anonymous fd.

When a statedump is generated and it's going to dump inode information, it takes the inode table lock and then each individual inode lock.

On the other side, when an anonymous fd is created, first it's acquired the lock from the inode, and then the lock from the inode table when the refcount of the inode is incremented.

This causes a deadlock because of reverse acquisition order of the locks.

This deadlock may not be exclusive of statedump, but there are few places where we take both locks at the same time, and for sure none of them does that in a so "massive" way.

Version-Release number of selected component (if applicable): mainline


How reproducible:

It's a race hard to reproduce.

Steps to Reproduce:
1.
2.
3.

Actual results:

Generation of statedump hangs and mount point becomes unresponsiveness.

Expected results:

Generation of statedump should complete successfully and mount point continue working normally.


Additional info:

Comment 1 Worker Ant 2019-07-04 11:26:20 UTC

REVIEW: https://review.gluster.org/22995 (core: fix deadlock between statedump and fd_anonymous()) posted (#1) for review on master by Xavi Hernandez

Comment 2 Worker Ant 2019-07-15 06:24:12 UTC

REVIEW: https://review.gluster.org/22995 (core: fix deadlock between statedump and fd_anonymous()) merged (#4) on master by Amar Tumballi