Bug 1302988

Description of problem:
On a 16 node setup with the vol configured as mentioned below, no demotions are seen from one of the hot tier bricks after continuous promotions and demotions running on the system for few hours.

Demotions are expected to work seamlessly from all nodes whenever files are eligible for demotion and other criteria such as max.mb, max.files etc are met.

#gluster v info krk-vol

Volume Name: krk-vol 
Type: Tier
Volume ID: 192655ce-4ef6-4ada-8e0c-6f137e2721e1
Status: Started
Number of Bricks: 36 
Transport-type: tcp  
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 6 x 2 = 12
Brick1: 10.70.37.101:/rhs/brick6/krkvol
Brick2: 10.70.37.69:/rhs/brick6/krkvol
Brick3: 10.70.37.60:/rhs/brick6/krkvol
Brick4: 10.70.37.120:/rhs/brick6/krkvol
Brick5: 10.70.37.202:/rhs/brick6/krkvol
Brick6: 10.70.37.195:/rhs/brick6/krkvol
Brick7: 10.70.35.44:/rhs/brick6/krkvol
Brick8: 10.70.35.231:/rhs/brick6/krkvol
Brick9: 10.70.35.176:/rhs/brick6/krkvol --> No demotion seen from this node
Brick10: 10.70.35.232:/rhs/brick6/krkvol --> No demotion seen from this node
Brick11: 10.70.35.173:/rhs/brick6/krkvol
Brick12: 10.70.35.163:/rhs/brick6/krkvol
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (8 + 4) = 24
Brick13: 10.70.35.176:/rhs/brick5/krkvol
Brick14: 10.70.35.232:/rhs/brick5/krkvol
Brick15: 10.70.35.173:/rhs/brick5/krkvol
Brick16: 10.70.35.163:/rhs/brick5/krkvol
Brick17: 10.70.37.101:/rhs/brick5/krkvol
Brick18: 10.70.37.69:/rhs/brick5/krkvol
Brick19: 10.70.37.60:/rhs/brick5/krkvol
Brick20: 10.70.37.120:/rhs/brick5/krkvol
Brick21: 10.70.37.202:/rhs/brick4/krkvol
Brick22: 10.70.37.195:/rhs/brick4/krkvol
Brick23: 10.70.35.155:/rhs/brick4/krkvol
Brick24: 10.70.35.222:/rhs/brick4/krkvol
Brick25: 10.70.35.108:/rhs/brick4/krkvol
Brick26: 10.70.35.44:/rhs/brick4/krkvol
Brick27: 10.70.35.89:/rhs/brick4/krkvol
Brick28: 10.70.35.231:/rhs/brick4/krkvol
Brick29: 10.70.35.176:/rhs/brick4/krkvol
Brick30: 10.70.35.232:/rhs/brick4/krkvol
Brick31: 10.70.35.173:/rhs/brick4/krkvol
Brick32: 10.70.35.163:/rhs/brick4/krkvol
Brick33: 10.70.37.101:/rhs/brick4/krkvol
Brick34: 10.70.37.69:/rhs/brick4/krkvol
Brick35: 10.70.37.60:/rhs/brick4/krkvol
Brick36: 10.70.37.120:/rhs/brick4/krkvol
Options Reconfigured:
cluster.read-freq-threshold: 1
cluster.write-freq-threshold: 1
features.record-counters: on
cluster.tier-demote-frequency: 600
performance.readdir-ahead: on
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: off
features.ctr-enabled: on
cluster.tier-mode: cache
performance.io-cache: off
performance.read-ahead: off
performance.open-behind: off
performance.write-behind: off
cluster.min-free-disk: 20
cluster.watermark-hi: 70
cluster.watermark-low: 55


Version-Release number of selected component (if applicable):
glusterfs-3.7.5-17.el7rhgs.x86_64

How reproducible:
Yet to determine

Steps to Reproduce:
1. create 16 node setup and create 2x(8+4) volume
2. enable quota and set limit usage to root directory
3. create two folders on this volume and fill data [folder1, folder2]
4. Attach 6x2 hot tier to this volume
5. create new files on another new folder [folder 3]
6. Now, continuously promote files from folder-1 making files eligible for promotion and have files on folder-2 and folder-3 idle making them eligible for demotion
7. Promote enough files so watermark can be breached
8. check if promotion & demotion is working as expected

Actual results:
 - Demotions from one replica pair (on files in folder-2 and folder-3) on hot tier doesn't happen
 - Demotion from other hot tier bricks work as expected. All files from folder-2 and folder-3 are demoted

Expected results:
 - Demotion should work on all bricks of hot tier

Additional info:
sosreport shall be attached. 
Files which were promoted always are files from folder - 'dd'. Files which were eligible but never demoted from a node were from files on folder - '1million-files' and 'new_files'.
Bricks from where files dont demote are - 10.70.35.176:/rhs/brick6/krkvol and 10.70.35.232:/rhs/brick6/krkvol