Bug 1366648
Summary: | [GSS] A hot tier brick becomes full, causing the entire volume to have issues and returns stale file handle and input/output error. | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Milind Changire <mchangir> | |
Component: | tiering | Assignee: | Milind Changire <mchangir> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | bugs <bugs> | |
Severity: | medium | Docs Contact: | ||
Priority: | medium | |||
Version: | mainline | CC: | amukherj, dlambrig, mchangir, mmalhotr, nbalacha, olim, rhs-bugs, storage-qa-internal | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.10.0 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | 1361759 | |||
: | 1394482 (view as bug list) | Environment: | ||
Last Closed: | 2017-03-06 17:22:10 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1361759, 1394482 |
Comment 1
Vijay Bellur
2016-08-12 14:53:23 UTC
REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#2) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#3) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#4) for review on master by Milind Changire (mchangir) Problem Description: Hot tier brick gets 100% full even when cluster.watermark-hi has been set to 90. Analysis: When IO is started on tiered volumes, there is no test to check if the usage of the brick has breached the hi-watermark. This causes the file to continue to grow beyond the hi-watermark and eventually consume the entire brick space if the IO continues without any checks. REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#5) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#6) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#7) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#8) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#9) for review on master by Milind Changire (mchangir) REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#10) for review on master by Milind Changire (mchangir) Oonkwee, We are indeed working/pushing hard to formulate a reasonable solution. The solution draft needs more discussions on the implementation approach. REVIEW: http://review.gluster.org/15158 (cluster/tier: handle fast demotions) posted (#11) for review on master by Milind Changire (mchangir) COMMIT: http://review.gluster.org/15158 committed in master by Dan Lambright (dlambrig) ------ commit 460016428cf27484c333227f534c2e2f73a37fb1 Author: Milind Changire <mchangir> Date: Sat Oct 15 10:49:19 2016 +0530 cluster/tier: handle fast demotions Demote files on priority if hi-watermark has been breached and continue to demote until the watermark drops below hi-watermark. Monitor watermark more frequently. Trigger demotion as soon as hi-watermark is breached. Add cluster.tier-emergency-demote-query-limit option to limit number of files returned from the database query for every iteration of tier_migrate_using_query_file(). If watermark hasn't dropped below hi-watermark during the first iteration, the next iteration will be triggered approximately 1 second after tier_demote() returns to the main tiering loop. Update changetimerecorder xlator to handle query for emergency demote mode. Add tier-ctr-interface.h: Move tier and ctr interface specific macros and struct definition from libglusterfs/src/gfdb/gfdb_data_store.h to new header libglusterfs/src/tier-ctr-interface.h Change-Id: If56af78c6c81d37529b9b6e65ae606ba5c99a811 BUG: 1366648 Signed-off-by: Milind Changire <mchangir> Reviewed-on: http://review.gluster.org/15158 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Dan Lambright <dlambrig> op-version needs fix REVIEW: http://review.gluster.org/15990 (cluster/tier: fix op-version for tier-query-limit) posted (#1) for review on master by Milind Changire (mchangir) COMMIT: http://review.gluster.org/15990 committed in master by Atin Mukherjee (amukherj) ------ commit 530453c78146e8ba4f13636e1dec1ea59849c783 Author: Milind Changire <mchangir> Date: Thu Dec 1 18:18:27 2016 +0530 cluster/tier: fix op-version for tier-query-limit Correct the op-version for tier-query-limit option from 3.9.0 to 3.9.1 Change-Id: I3a52a94c2708a97c18377e945d559a51d8025c41 BUG: 1366648 Signed-off-by: Milind Changire <mchangir> Reviewed-on: http://review.gluster.org/15990 Reviewed-by: Dan Lambright <dlambrig> Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Atin Mukherjee <amukherj> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report. glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html [2] https://www.gluster.org/pipermail/gluster-users/ |