Red Hat Bugzilla – Bug 1296016
[Tiering]: Hot tier disk usage reaches 100% as no demotions are seen in the system
Last modified: 2016-09-17 11:39:52 EDT
Description of problem:
On a 8 node gluster cluster, with dist-rep cold and hot tier, no demotions of files are seen and as a result, we end up utilizing 100% of hot tier's capacity.
No promotions were triggered throughout this test. Hot tier's capacity - 200G and cold tier's capacity - 2 TB. 1 GB files were written using dd using a for loop, hence IOs were serial. No demotions were seen although there was ample time for demotion and files to be selected for demotion before filling up hot tier.
This must be a regression as this behavior was not seen earlier.
volume used for test - dist-rep-coldtier
quota enabled and / directory set to 1 TB
watermarks were set at 50% and 70%
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Create a dist-repl cold tier
2. Enable quota and set limit-usage to / directory
3. Attach a dist-repl hot tier
2. set watermark levels at 50% and 70%
3. create 1Gb files serially
No demotions are seen and hot tier disk usage reaches 100%
Demotions of files which were written earlier should start after crossing low watermark as demotion freq is set to default 120 sec. Any existing files which were written before 120 sec after crossing low watermark should be considered for demotion.
sosreport shall be attached shortly
sosreports are available here
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1296016/
This issue is no more seen with the GA build of RHGS 3.1.2. Following test was done to validate.
1) create a dist-rep volume
2) enable quota and set limit to / directory as 10GB
3) attached tier
4) set high watermark to 50% (i.e., 5GB) and low watermark to 20% (i.e., 20GB)
4) mounted the volume from client and used dd to create ten 1GB files
5) Files were written on hot tier
Once high watermark levels had crossed, file got demoted from hot tier to cold tier.
This issue is not seen in build glusterfs-3.7.5-19.el7rhgs.x86_64, Moving the bz to verified.