Description of problem: On a 16 node setup with the vol configured as mentioned below, no demotions are seen from one of the hot tier bricks after continuous promotions and demotions running on the system for few hours. Demotions are expected to work seamlessly from all nodes whenever files are eligible for demotion and other criteria such as max.mb, max.files etc are met. #gluster v info krk-vol Volume Name: krk-vol Type: Tier Volume ID: 192655ce-4ef6-4ada-8e0c-6f137e2721e1 Status: Started Number of Bricks: 36 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 6 x 2 = 12 Brick1: 10.70.37.101:/rhs/brick6/krkvol Brick2: 10.70.37.69:/rhs/brick6/krkvol Brick3: 10.70.37.60:/rhs/brick6/krkvol Brick4: 10.70.37.120:/rhs/brick6/krkvol Brick5: 10.70.37.202:/rhs/brick6/krkvol Brick6: 10.70.37.195:/rhs/brick6/krkvol Brick7: 10.70.35.44:/rhs/brick6/krkvol Brick8: 10.70.35.231:/rhs/brick6/krkvol Brick9: 10.70.35.176:/rhs/brick6/krkvol --> No demotion seen from this node Brick10: 10.70.35.232:/rhs/brick6/krkvol --> No demotion seen from this node Brick11: 10.70.35.173:/rhs/brick6/krkvol Brick12: 10.70.35.163:/rhs/brick6/krkvol Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (8 + 4) = 24 Brick13: 10.70.35.176:/rhs/brick5/krkvol Brick14: 10.70.35.232:/rhs/brick5/krkvol Brick15: 10.70.35.173:/rhs/brick5/krkvol Brick16: 10.70.35.163:/rhs/brick5/krkvol Brick17: 10.70.37.101:/rhs/brick5/krkvol Brick18: 10.70.37.69:/rhs/brick5/krkvol Brick19: 10.70.37.60:/rhs/brick5/krkvol Brick20: 10.70.37.120:/rhs/brick5/krkvol Brick21: 10.70.37.202:/rhs/brick4/krkvol Brick22: 10.70.37.195:/rhs/brick4/krkvol Brick23: 10.70.35.155:/rhs/brick4/krkvol Brick24: 10.70.35.222:/rhs/brick4/krkvol Brick25: 10.70.35.108:/rhs/brick4/krkvol Brick26: 10.70.35.44:/rhs/brick4/krkvol Brick27: 10.70.35.89:/rhs/brick4/krkvol Brick28: 10.70.35.231:/rhs/brick4/krkvol Brick29: 10.70.35.176:/rhs/brick4/krkvol Brick30: 10.70.35.232:/rhs/brick4/krkvol Brick31: 10.70.35.173:/rhs/brick4/krkvol Brick32: 10.70.35.163:/rhs/brick4/krkvol Brick33: 10.70.37.101:/rhs/brick4/krkvol Brick34: 10.70.37.69:/rhs/brick4/krkvol Brick35: 10.70.37.60:/rhs/brick4/krkvol Brick36: 10.70.37.120:/rhs/brick4/krkvol Options Reconfigured: cluster.read-freq-threshold: 1 cluster.write-freq-threshold: 1 features.record-counters: on cluster.tier-demote-frequency: 600 performance.readdir-ahead: on features.quota: on features.inode-quota: on features.quota-deem-statfs: off features.ctr-enabled: on cluster.tier-mode: cache performance.io-cache: off performance.read-ahead: off performance.open-behind: off performance.write-behind: off cluster.min-free-disk: 20 cluster.watermark-hi: 70 cluster.watermark-low: 55 Version-Release number of selected component (if applicable): glusterfs-3.7.5-17.el7rhgs.x86_64 How reproducible: Yet to determine Steps to Reproduce: 1. create 16 node setup and create 2x(8+4) volume 2. enable quota and set limit usage to root directory 3. create two folders on this volume and fill data [folder1, folder2] 4. Attach 6x2 hot tier to this volume 5. create new files on another new folder [folder 3] 6. Now, continuously promote files from folder-1 making files eligible for promotion and have files on folder-2 and folder-3 idle making them eligible for demotion 7. Promote enough files so watermark can be breached 8. check if promotion & demotion is working as expected Actual results: - Demotions from one replica pair (on files in folder-2 and folder-3) on hot tier doesn't happen - Demotion from other hot tier bricks work as expected. All files from folder-2 and folder-3 are demoted Expected results: - Demotion should work on all bricks of hot tier Additional info: sosreport shall be attached. Files which were promoted always are files from folder - 'dd'. Files which were eligible but never demoted from a node were from files on folder - '1million-files' and 'new_files'. Bricks from where files dont demote are - 10.70.35.176:/rhs/brick6/krkvol and 10.70.35.232:/rhs/brick6/krkvol
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1295299/
Thank you for your bug report. We are no longer working on any improvements for Tier. This bug will be set to CLOSED WONTFIX to reflect this. Please reopen if the rfe is deemed critical.