Description of problem:

Once high watermark is hit, demotions are too aggressive and demotion continues even after low water mark level. Demotions continued till %used reached up to 36%. 

Demotions should stop once low water mark is reached. 

Total Hot tier size - 40G
high water mark - 36G
low water mark - 30G

After demotions had stopped, hot tier was at 14.3G

<<<<<<<<<Hot tier bricks after demotions stopped>>>>>>>>>>>

[root@dhcp42-47 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G  2.3G   18G  12% /rhs/brick4
[root@dhcp43-19 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G  2.3G   18G  12% /rhs/brick4


[root@dhcp42-177 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G   12G  8.4G  59% /rhs/brick4
[root@dhcp42-10 ~]# df -h /rhs/brick4/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg4-RHS_lv4   20G   12G  8.4G  59% /rhs/brick4

<<<<<<<<<<<<<<cold tier bricks after demotions >>>>>>>>>>>>

[root@dhcp42-47 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp43-19 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-177 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-10 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp43-140 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-87 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   16G   35G  32% /rhs/brick1
[root@dhcp42-228 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   19G   32G  37% /rhs/brick1
[root@dhcp42-183 ~]# df -h /rhs/brick1/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg1-RHS_lv1   50G   19G   32G  37% /rhs/brick1
[root@dhcp42-47 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2
[root@dhcp43-19 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2
[root@dhcp42-177 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2
[root@dhcp42-10 ~]# df -h /rhs/brick2/leg1/
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/RHS_vg2-RHS_lv2   50G   19G   32G  37% /rhs/brick2



# gluster vol info

Volume Name: tier-vol-01
Type: Tier
Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60
Status: Started
Number of Bricks: 16
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.42.10:/rhs/brick4/leg1
Brick2: 10.70.42.177:/rhs/brick4/leg1
Brick3: 10.70.43.19:/rhs/brick4/leg1
Brick4: 10.70.42.47:/rhs/brick4/leg1
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick5: 10.70.42.47:/rhs/brick1/leg1
Brick6: 10.70.43.19:/rhs/brick1/leg1
Brick7: 10.70.42.177:/rhs/brick1/leg1
Brick8: 10.70.42.10:/rhs/brick1/leg1
Brick9: 10.70.43.140:/rhs/brick1/leg1
Brick10: 10.70.42.87:/rhs/brick1/leg1
Brick11: 10.70.42.228:/rhs/brick1/leg1
Brick12: 10.70.42.183:/rhs/brick1/leg1
Brick13: 10.70.42.47:/rhs/brick2/leg1
Brick14: 10.70.43.19:/rhs/brick2/leg1
Brick15: 10.70.42.177:/rhs/brick2/leg1
Brick16: 10.70.42.10:/rhs/brick2/leg1
Options Reconfigured:
cluster.tier-promote-frequency: 120
cluster.tier-mode: cache
cluster.tier-demote-frequency: 120
features.record-counters: on
cluster.watermark-hi: 95
cluster.watermark-low: 75
features.ctr-enabled: on
performance.readdir-ahead: on


Version-Release number of selected component (if applicable):
glusterfs-3.7.5-11.el7rhgs.x86_64

How reproducible:
Yet to determine if this is 100%. Have seen this behavior at least twice

Steps to Reproduce:
1. Configure a tiered volume
2. Make configuration changes for the volume as shown in vol info o/p above
3. Write new files and promote old files from cold tier so that high water mark is reached
3. Once high water mark is reached, stop all IOs

Actual results:

 - Too many demotions are triggered such that hot tier %used is reduced till 36% whereas low watermark is set at 75%

Expected results:
Demotions should stop when low watermark is hit

Additional info:
sosreports will be attached shortly