Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1727068

Summary: Deadlock when generating statedumps
Product: [Community] GlusterFS Reporter: Xavi Hernandez <jahernan>
Component: coreAssignee: Xavi Hernandez <jahernan>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: mainlineCC: bugs, pasik
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1729950 1729952 (view as bug list) Environment:
Last Closed: 2019-07-15 06:24:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1722209, 1729950, 1729952    

Description Xavi Hernandez 2019-07-04 11:12:29 UTC 
Description of problem:

There seems to be a deadlock between statedump generation and creation of an anonymous fd.

When a statedump is generated and it's going to dump inode information, it takes the inode table lock and then each individual inode lock.

On the other side, when an anonymous fd is created, first it's acquired the lock from the inode, and then the lock from the inode table when the refcount of the inode is incremented.

This causes a deadlock because of reverse acquisition order of the locks.

This deadlock may not be exclusive of statedump, but there are few places where we take both locks at the same time, and for sure none of them does that in a so "massive" way.

Version-Release number of selected component (if applicable): mainline


How reproducible:

It's a race hard to reproduce.

Steps to Reproduce:
1.
2.
3.

Actual results:

Generation of statedump hangs and mount point becomes unresponsiveness.

Expected results:

Generation of statedump should complete successfully and mount point continue working normally.


Additional info:
Comment 1 Worker Ant 2019-07-04 11:26:20 UTC 
REVIEW: https://review.gluster.org/22995 (core: fix deadlock between statedump and fd_anonymous()) posted (#1) for review on master by Xavi Hernandez
Comment 2 Worker Ant 2019-07-15 06:24:12 UTC 
REVIEW: https://review.gluster.org/22995 (core: fix deadlock between statedump and fd_anonymous()) merged (#4) on master by Amar Tumballi