Bug 1296016

Summary: [Tiering]: Hot tier disk usage reaches 100% as no demotions are seen in the system
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: krishnaram Karthick <kramdoss>
Component: tierAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED CURRENTRELEASE QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: byarlaga, kramdoss, nbalacha, rhs-bugs, smohan, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: tier-migration
Fixed In Version: glusterfs-3.7.5-19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-28 06:23:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1300246    
Bug Blocks:    

Description krishnaram Karthick 2016-01-06 07:24:08 UTC 
Description of problem:
On a 8 node gluster cluster, with dist-rep cold and hot tier, no demotions of files are seen and as a result, we end up utilizing 100% of hot tier's capacity.
No promotions were triggered throughout this test. Hot tier's capacity - 200G and cold tier's capacity - 2 TB. 1 GB files were written using dd using a for loop, hence IOs were serial. No demotions were seen although there was ample time for demotion and files to be selected for demotion before filling up hot tier. 
This must be a regression as this behavior was not seen earlier.

volume used for test - dist-rep-coldtier
options reconfigured:
quota enabled and / directory set to 1 TB
watermarks were set at 50% and 70%

Version-Release number of selected component (if applicable):
glusterfs-3.7.5-13.el7rhgs.x86_64

How reproducible:
2/2

Steps to Reproduce:
1. Create a dist-repl cold tier
2. Enable quota and set limit-usage to / directory
3. Attach a dist-repl hot tier
2. set watermark levels at 50% and 70% 
3. create 1Gb files serially

Actual results:
No demotions are seen and hot tier disk usage reaches 100%

Expected results:
Demotions of files which were written earlier should start after crossing low watermark as demotion freq is set to default 120 sec. Any existing files which were written before 120 sec after crossing low watermark should be considered for demotion.

Additional info:
sosreport shall be attached shortly
Comment 2 krishnaram Karthick 2016-01-06 07:34:35 UTC 
sosreports are available here
Comment 3 krishnaram Karthick 2016-01-06 08:07:44 UTC 
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1296016/
Comment 12 krishnaram Karthick 2016-03-07 16:23:04 UTC 
This issue is no more seen with the GA build of RHGS 3.1.2. Following test was done to validate.

1) create a dist-rep volume
2) enable quota and set limit to / directory as 10GB
3) attached tier
4) set high watermark to 50% (i.e., 5GB) and low watermark to 20% (i.e., 20GB)
4) mounted the volume from client and used dd to create ten 1GB files
5) Files were written on hot tier

Once high watermark levels had crossed, file got demoted from hot tier to cold tier.

This issue is not seen in build glusterfs-3.7.5-19.el7rhgs.x86_64, Moving the bz to verified.