Bug 1291549 - [tiering]: When high watermark is hit, demotions are too aggressive that it demotes files way beyond low water mark
[tiering]: When high watermark is hit, demotions are too aggressive that it d...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Dan Lambright
nchilaka
tier-migration
: ZStream
Depends On: 1291969
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-15 01:20 EST by krishnaram Karthick
Modified: 2017-03-25 10:23 EDT (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-03-25 10:23:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description krishnaram Karthick 2015-12-15 01:20:32 EST
Description of problem:

Once high watermark is hit, demotions are too aggressive and demotion continues even after low water mark level. Demotions continued till %used reached up to 36%. 

Demotions should stop once low water mark is reached. 

Total Hot tier size - 40G
high water mark - 36G
low water mark - 30G

After demotions had stopped, hot tier was at 14.3G

<<<<<<<<<Hot tier bricks after demotions stopped>>>>>>>>>>>

[root@dhcp42-47 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G  2.3G   18G  12% /rhs/brick4
[root@dhcp43-19 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G  2.3G   18G  12% /rhs/brick4


[root@dhcp42-177 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G   12G  8.4G  59% /rhs/brick4
[root@dhcp42-10 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G   12G  8.4G  59% /rhs/brick4

<<<<<<<<<<<<<<cold tier bricks after demotions >>>>>>>>>>>>

[root@dhcp42-47 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp43-19 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-177 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-10 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp43-140 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-87 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-228 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   19G   32G  37% /rhs/brick1
[root@dhcp42-183 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   19G   32G  37% /rhs/brick1
[root@dhcp42-47 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2
[root@dhcp43-19 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2
[root@dhcp42-177 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2
[root@dhcp42-10 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2



# gluster vol info

Volume Name: tier-vol-01
Type: Tier
Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60
Status: Started
Number of Bricks: 16
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.42.10:/rhs/brick4/leg1
Brick2: 10.70.42.177:/rhs/brick4/leg1
Brick3: 10.70.43.19:/rhs/brick4/leg1
Brick4: 10.70.42.47:/rhs/brick4/leg1
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick5: 10.70.42.47:/rhs/brick1/leg1
Brick6: 10.70.43.19:/rhs/brick1/leg1
Brick7: 10.70.42.177:/rhs/brick1/leg1
Brick8: 10.70.42.10:/rhs/brick1/leg1
Brick9: 10.70.43.140:/rhs/brick1/leg1
Brick10: 10.70.42.87:/rhs/brick1/leg1
Brick11: 10.70.42.228:/rhs/brick1/leg1
Brick12: 10.70.42.183:/rhs/brick1/leg1
Brick13: 10.70.42.47:/rhs/brick2/leg1
Brick14: 10.70.43.19:/rhs/brick2/leg1
Brick15: 10.70.42.177:/rhs/brick2/leg1
Brick16: 10.70.42.10:/rhs/brick2/leg1
Options Reconfigured:
cluster.tier-promote-frequency: 120
cluster.tier-mode: cache
cluster.tier-demote-frequency: 120
features.record-counters: on
cluster.watermark-hi: 95
cluster.watermark-low: 75
features.ctr-enabled: on
performance.readdir-ahead: on


Version-Release number of selected component (if applicable):
glusterfs-3.7.5-11.el7rhgs.x86_64

How reproducible:
Yet to determine if this is 100%. Have seen this behavior at least twice

Steps to Reproduce:
1. Configure a tiered volume
2. Make configuration changes for the volume as shown in vol info o/p above
3. Write new files and promote old files from cold tier so that high water mark is reached
3. Once high water mark is reached, stop all IOs

Actual results:

 - Too many demotions are triggered such that hot tier %used is reduced till 36% whereas low watermark is set at 75%

Expected results:
Demotions should stop when low watermark is hit

Additional info:
sosreports will be attached shortly
Comment 2 krishnaram Karthick 2015-12-15 01:26:39 EST
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291549/
Comment 7 krishnaram Karthick 2016-04-29 04:38:58 EDT
verified the bug in 3.1.2 and demotion works as expected when high watermarks are crossed. Moving the bug to verified.

Note You need to log in before you can comment on or make changes to this bug.