Bug 843003
| Summary: | call_bail of a frame in glusterd might lead to stale locks in the cluster | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra Bhat <rabhat> | |
| Component: | glusterd | Assignee: | Kaushal <kaushal> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | high | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 3.3.0 | CC: | gluster-bugs, kaushal, mailbox | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.4.0 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 844682 (view as bug list) | Environment: | ||
| Last Closed: | 2013-07-24 17:54:07 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 844682, 844803 | |||
|
Description
Raghavendra Bhat
2012-07-25 09:23:02 UTC
CHANGE: http://review.gluster.com/3803 (rpc: Reduce frame-timeout for glusterd connections) merged in master by Vijay Bellur (vbellur) CHANGE: http://review.gluster.com/3812 (rpc: Reduce frame-timeout for glusterd connections) merged in release-3.3 by Vijay Bellur (vbellur) These are some comments on this problem and why it was solved this way. gluster uses different timeouts to make sure that a hung operation or an operation taking a long time to complete doesn't block other operations forever. The default timeout used to be 30min for glusterd and 2min for cli. This meant that cli would stop waiting for glusterd to reply and exit after 2 minutes, whereas glusterd would still keep waiting for 30 mins before cancelling the operation. Glusterd uses cluster wide locks to co-ordinate with other glusterds and keep consistency. Most of the volume operations use these locks. These locks are acquired at the beginning of the opeartion and released once the operation finishes. This would lead to problems with volume operations which take a long time to complete (like some of the 'volume status' operations). cli used to timeout after 2 mins. But glusterd would still continue to hold the locks till the operations completed or the timeout of 30 mins was reached (call_bail happens on timeout in glusterd). This lead to a potentially long window of 28 mins before any other operation could be initiated from cli. Most cli commands in this window will fail because glusterd will be unable to acquire the cluster lock. This is the same in the case of the bug report above. The truncated logs don't give the full picture. By going through the full logs we found that normal operations resumed after 30 mins as a call_bail leads to the locks being released, and the operations were only failing in this 30min window. The failure to obtain locks is because of concurrent execution of commands from different peers, not because of call_bail as stated in the bug title. The fix for this is to reduce the glusterd timeout, so that the window isn't as long hence leads to lesser waiting time before being able to successfully do volume operations once again. |