Bug 1291969
Summary: | [Tiering]: When files are heated continuously, promotions are too aggressive that it promotes files way beyond high water mark | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | krishnaram Karthick <kramdoss> | |
Component: | tier | Assignee: | Bug Updates Notification Mailing List <rhs-bugs> | |
Status: | CLOSED ERRATA | QA Contact: | krishnaram Karthick <kramdoss> | |
Severity: | urgent | Docs Contact: | ||
Priority: | urgent | |||
Version: | rhgs-3.1 | CC: | asrivast, byarlaga, dlambrig, rcyriac, rhs-bugs, sankarshan, storage-qa-internal | |
Target Milestone: | --- | Keywords: | ZStream | |
Target Release: | RHGS 3.1.2 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.7.5-15 | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1293932 (view as bug list) | Environment: | ||
Last Closed: | 2016-03-01 06:04:07 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1291549, 1293932 |
Description
krishnaram Karthick
2015-12-16 05:16:32 UTC
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291969/ Hi Karthick. What is the file size you are using? File sizes used were in the range of 100Mb for the initial test when the total hot tier was 20Gb. Each node may independently pick a file to promote. There are 12 nodes on the cold tier, so no more than 1.2G should be moved in parallel in a single direction. This could leave to an overshoot of 1.2G (I think), but you see more than that. What each node should be doing is (1) checking free space on the entire hot tier, abort if there is not enough space to move a particular 100Mb file (2) if there is space, move the file, (3) back to (1). On code inspection, it looks like step (1) is improperly implemented. I'll see if I can recreate this and write a fix if my inspection proves to be correct. Patch 64888 submitted for this problem. Verified the fix on build glusterfs-3.7.5-15 Following tests were performed to verify the fix. Test 1: 1) Created a dist-rep volume (300Gb), enabled quota 2) Fuse mounted the volume and created 1000 files of 10Mb 3) Attached a replicated tier brick(50Gb) 4) Modified watermark to 45% and 70% 5) Heated all 1000 files created in step 2. No new files were written Result: Disk usage hit 70% and never increased beyond Test 2: After test1, increased watermark to 80% Result: Disk usage hit 80% and never increased beyond Test 3: After test 2, decreased watermark to 70% Result: Disk usage reached 70% and never increased beyond Test 4: stopped heating files and left the system idle. Result: Disk usage reached 45% and never decreased beyond Test 5: Repeated test 1 and test 4 with promotion cycle at 300s and demotion cycle at 1800s. Results were same. Moving the bug to verified state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html |