Bug 1346719
Summary: | [Disperse] dd + rm + ls lead to IO hang | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Ashish Pandey <aspandey> | |
Component: | disperse | Assignee: | Pranith Kumar K <pkarampu> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
Severity: | high | Docs Contact: | ||
Priority: | medium | |||
Version: | mainline | CC: | amukherj, bugs, hgowtham, jahernan, pkarampu, ravishankar | |
Target Milestone: | --- | Keywords: | Triaged | |
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.9.0 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1361519 1362420 1371397 1373392 (view as bug list) | Environment: | ||
Last Closed: | 2017-03-27 18:20:24 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1361519, 1362420, 1371397, 1373392 |
Description
Ashish Pandey
2016-06-15 08:47:02 UTC
statedump shows some blocked inodelk - [conn.1.bound_xl./brick/gluster/vol-1.active.1] gfid=00000000-0000-0000-0000-000000000001 nlookup=3 fd-count=3 ref=1 ia_type=2 [xlator.features.locks.vol-locks.inode] path=/ mandatory=0 inodelk-count=3 lock-dump.domain.domain=dht.layout.heal lock-dump.domain.domain=vol-disperse-0:self-heal lock-dump.domain.domain=vol-disperse-0 inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 3327, owner=dc710738fd7e0000, client=0x7f283c1a7b00, connection-id=apandey-15766-2016/06/15-07:59:38:894408-vol-client-0-0-0, blocked at 2016-06-15 08:02:13, granted at 2016-06-15 08:02:13 inodelk.inodelk[1](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid = 22451, owner=cc338ae8f07f0000, client=0x7f2834006660, connection-id=apandey-13531-2016/06/15-07:58:50:360055-vol-client-0-0-0, blocked at 2016-06-15 08:02:13 inodelk.inodelk[2](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid = 22530, owner=6cd51d48da7f0000, client=0x7f28342db820, connection-id=apandey-19856-2016/06/15-08:01:05:258794-vol-client-0-0-0, blocked at 2016-06-15 08:02:22 Just observed that option disperse.eager-lock has come to rescue- Setting disperse.eager-lock to off started IO's and ls -lR command. gluster v set vol disperse.eager-lock off REVIEW: http://review.gluster.org/15309 (cluster/ec: Use locks for opendir) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu) COMMIT: http://review.gluster.org/15309 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit f013335400d033a9677797377b90b968803135f4 Author: Pranith Kumar K <pkarampu> Date: Wed Aug 24 21:01:05 2016 +0530 cluster/ec: Use locks for opendir Problem: In some cases we see that readdir keeps winding to the brick that doesn't have any blocked locks i.e. first brick. This is leading to the client assuming that there are no blocking locks on the inode so it won't give away the lock. Other clients end up blocked on the lock as if the command hung. Fix: Proper way to fix this issue is to use infra present in http://review.gluster.org/14736 This is a stop gap fix where we start taking inodelks in opendir which goes to all the bricks, this will detect if there is any contention. BUG: 1346719 Change-Id: I91109107a26f6535b945ac476338e9f21dc31eb9 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/15309 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Ashish Pandey <aspandey> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report. glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html [2] https://www.gluster.org/pipermail/gluster-users/ |