Bug 1727068 - Deadlock when generating statedumps
Summary: Deadlock when generating statedumps
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: core
Version: mainline
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
Assignee: Xavi Hernandez
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1722209 1729950 1729952
TreeView+ depends on / blocked
 
Reported: 2019-07-04 11:12 UTC by Xavi Hernandez
Modified: 2019-07-15 12:07 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1729950 1729952 (view as bug list)
Environment:
Last Closed: 2019-07-15 06:24:12 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Gluster.org Gerrit 22995 0 None Merged core: fix deadlock between statedump and fd_anonymous() 2019-07-15 06:24:11 UTC

Description Xavi Hernandez 2019-07-04 11:12:29 UTC
Description of problem:

There seems to be a deadlock between statedump generation and creation of an anonymous fd.

When a statedump is generated and it's going to dump inode information, it takes the inode table lock and then each individual inode lock.

On the other side, when an anonymous fd is created, first it's acquired the lock from the inode, and then the lock from the inode table when the refcount of the inode is incremented.

This causes a deadlock because of reverse acquisition order of the locks.

This deadlock may not be exclusive of statedump, but there are few places where we take both locks at the same time, and for sure none of them does that in a so "massive" way.

Version-Release number of selected component (if applicable): mainline


How reproducible:

It's a race hard to reproduce.

Steps to Reproduce:
1.
2.
3.

Actual results:

Generation of statedump hangs and mount point becomes unresponsiveness.

Expected results:

Generation of statedump should complete successfully and mount point continue working normally.


Additional info:

Comment 1 Worker Ant 2019-07-04 11:26:20 UTC
REVIEW: https://review.gluster.org/22995 (core: fix deadlock between statedump and fd_anonymous()) posted (#1) for review on master by Xavi Hernandez

Comment 2 Worker Ant 2019-07-15 06:24:12 UTC
REVIEW: https://review.gluster.org/22995 (core: fix deadlock between statedump and fd_anonymous()) merged (#4) on master by Amar Tumballi


Note You need to log in before you can comment on or make changes to this bug.