Bug 1296016 - [Tiering]: Hot tier disk usage reaches 100% as no demotions are seen in the system
[Tiering]: Hot tier disk usage reaches 100% as no demotions are seen in the s...
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Nithya Balachandran
: ZStream
Depends On: 1300246
  Show dependency treegraph
Reported: 2016-01-06 02:24 EST by krishnaram Karthick
Modified: 2016-09-17 11:39 EDT (History)
6 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-07-28 02:23:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description krishnaram Karthick 2016-01-06 02:24:08 EST
Description of problem:
On a 8 node gluster cluster, with dist-rep cold and hot tier, no demotions of files are seen and as a result, we end up utilizing 100% of hot tier's capacity.
No promotions were triggered throughout this test. Hot tier's capacity - 200G and cold tier's capacity - 2 TB. 1 GB files were written using dd using a for loop, hence IOs were serial. No demotions were seen although there was ample time for demotion and files to be selected for demotion before filling up hot tier. 
This must be a regression as this behavior was not seen earlier.

volume used for test - dist-rep-coldtier
options reconfigured:
quota enabled and / directory set to 1 TB
watermarks were set at 50% and 70%

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Create a dist-repl cold tier
2. Enable quota and set limit-usage to / directory
3. Attach a dist-repl hot tier
2. set watermark levels at 50% and 70% 
3. create 1Gb files serially

Actual results:
No demotions are seen and hot tier disk usage reaches 100%

Expected results:
Demotions of files which were written earlier should start after crossing low watermark as demotion freq is set to default 120 sec. Any existing files which were written before 120 sec after crossing low watermark should be considered for demotion.

Additional info:
sosreport shall be attached shortly
Comment 2 krishnaram Karthick 2016-01-06 02:34:35 EST
sosreports are available here
Comment 3 krishnaram Karthick 2016-01-06 03:07:44 EST
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1296016/
Comment 12 krishnaram Karthick 2016-03-07 11:23:04 EST
This issue is no more seen with the GA build of RHGS 3.1.2. Following test was done to validate.

1) create a dist-rep volume
2) enable quota and set limit to / directory as 10GB
3) attached tier
4) set high watermark to 50% (i.e., 5GB) and low watermark to 20% (i.e., 20GB)
4) mounted the volume from client and used dd to create ten 1GB files
5) Files were written on hot tier

Once high watermark levels had crossed, file got demoted from hot tier to cold tier.

This issue is not seen in build glusterfs-3.7.5-19.el7rhgs.x86_64, Moving the bz to verified.

Note You need to log in before you can comment on or make changes to this bug.