Description of problem: Files are not promoted based on the values set in 'cluster.tier-max-files'. When cluster.tier-max-files was set to '2' and 5 files were heated, all 5 files were promoted instead of 2 files. Version-Release number of selected component (if applicable): glusterfs-3.7.5-11.el7rhgs.x86_64 How reproducible: 100% Steps to Reproduce: 1. set 'cluster.tier-max-files' to 2 2. Heat 5 files from cold tier Actual results: All 5 files are promoted Expected results: only 2 files have to be promoted Additional info: sosreports will be attached shortly. Files that were used for this test are /mnt/tier-vol-01/file-99{1..8}
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291152/
o/p of gluster vol info # gluster vol info Volume Name: tier-vol-01 Type: Tier Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60 Status: Started Number of Bricks: 16 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: 10.70.42.10:/rhs/brick4/leg1 Brick2: 10.70.42.177:/rhs/brick4/leg1 Brick3: 10.70.43.19:/rhs/brick4/leg1 Brick4: 10.70.42.47:/rhs/brick4/leg1 Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (4 + 2) = 12 Brick5: 10.70.42.47:/rhs/brick1/leg1 Brick6: 10.70.43.19:/rhs/brick1/leg1 Brick7: 10.70.42.177:/rhs/brick1/leg1 Brick8: 10.70.42.10:/rhs/brick1/leg1 Brick9: 10.70.43.140:/rhs/brick1/leg1 Brick10: 10.70.42.87:/rhs/brick1/leg1 Brick11: 10.70.42.228:/rhs/brick1/leg1 Brick12: 10.70.42.183:/rhs/brick1/leg1 Brick13: 10.70.42.47:/rhs/brick2/leg1 Brick14: 10.70.43.19:/rhs/brick2/leg1 Brick15: 10.70.42.177:/rhs/brick2/leg1 Brick16: 10.70.42.10:/rhs/brick2/leg1 Options Reconfigured: cluster.tier-max-files: 2 cluster.tier-promote-frequency: 120 cluster.tier-mode: cache cluster.tier-demote-frequency: 120 features.record-counters: on cluster.watermark-hi: 10 cluster.watermark-low: 5 features.ctr-enabled: on performance.readdir-ahead: on
You are running 4 nodes on the hot tier. The limit tier-max-files is effective per node. Because files are distributed across the 4 nodes, the scenario you describe is legal as each node will individually not cross the limit even though the sum of all the nodes will. Suggest to retry with just a single node on the hot tier.
When hot tier is made of 1 node and cluster.tier-max-files is set to 1, more than 1 file gets promoted in a single cycle. When 9 files were heated - On my first attempt 2 files were promoted - second attempt 2 files were promoted - third attempt 3 files were promoted - Fourth attempt 4 files were promoted As a whole, the behavior is not consistent, but it is pretty sure that the value set for 'cluster.tier-max-files' isn't honored. # gluster vol info Volume Name: tier-vol-01 Type: Tier Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60 Status: Started Number of Bricks: 13 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.70.42.47:/rhs/brick4/leg2 Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (4 + 2) = 12 Brick2: 10.70.42.47:/rhs/brick1/leg1 Brick3: 10.70.43.19:/rhs/brick1/leg1 Brick4: 10.70.42.177:/rhs/brick1/leg1 Brick5: 10.70.42.10:/rhs/brick1/leg1 Brick6: 10.70.43.140:/rhs/brick1/leg1 Brick7: 10.70.42.87:/rhs/brick1/leg1 Brick8: 10.70.42.228:/rhs/brick1/leg1 Brick9: 10.70.42.183:/rhs/brick1/leg1 Brick10: 10.70.42.47:/rhs/brick2/leg1 Brick11: 10.70.43.19:/rhs/brick2/leg1 Brick12: 10.70.42.177:/rhs/brick2/leg1 Brick13: 10.70.42.10:/rhs/brick2/leg1 Options Reconfigured: cluster.tier-max-files: 1 cluster.tier-mode: cache features.ctr-enabled: on cluster.tier-promote-frequency: 120 cluster.tier-demote-frequency: 120 features.record-counters: on cluster.watermark-hi: 60 cluster.watermark-low: 30 performance.readdir-ahead: on Moreover, as per the definition of 'cluster.tier-max-files' in help, we don't define that this setting is based on number of nodes. It is easy to assume that this setting applies as a whole to a volume. We'll have to modify our help section to reflect the actual behavior. Option: cluster.tier-max-files Default Value: 5000 Description: The maximum number of files that may be migrated in any direction in a given cycle. sosreport for today's attempt to reproduce will be attached shortly.
Files that got promoted when files-{981..999} were heated up are pasted below. [root@dhcp42-47 tmp]# ll -lh /rhs/brick4/leg2/dd/ total 1.1G -rw-r--r--. 2 root root 102M Dec 15 15:16 file-981 -rw-r--r--. 2 root root 102M Dec 15 15:16 file-983 -rw-r--r--. 2 root root 102M Dec 15 15:16 file-985 -rw-r--r--. 2 root root 102M Dec 15 15:16 file-987 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-992 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-993 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-994 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-995 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-996 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-997 -rw-r--r--. 2 root root 102M Dec 15 15:11 file-999
sosreports are available here
You said you tried with one node, but your volume info shows 10.70.42.47, 10.70.43.19, etc. ; Each IP address would be different node. Is "tier-vol-01" the volume you tried? If not, can you paste what was? I do not see any link to sos reports in comment 7, can you post that as well? We may be allowing migration of max-migrate-files+1 rather than max-migrate_files. Your comment in the help message w.r.t documentation is valid and could be fixed using this bug as a vehicle.
The volume under test was tier-vol-01. Although the test system has 8 nodes, only one brick out of all 8 nodes is used as hot tier. i.e., Brick1: 10.70.42.47:/rhs/brick4/leg2. Rest of the bricks seen in vol info o/p are cold tier. sosreports --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291152/reproduced_on_15dec/
The limit is on the node in the migration source, not the destination. You are doing promotions, so up to 6 nodes may be source nodes as your cold tier in a given cycle.
http://review.gluster.org/12984
Verified the fix in build glusterfs-server-3.7.5-13.el7rhgs.x86_64 Steps followed to verify: 1) configured a 1 brick gluster volume 2) created 100 files 3) Attached a 1 brick hot tier 4) set cluster.tier-max-files to 50 5) heated all 100 files from the fuse mount by writing data 6) only 50 files were promoted Repeated the test with 2 brick distributed cold tier and 1 brick hot tier. steps: 1) configured a 2 brick gluster volume 2) created 100 files 3) Attached a 1 brick hot tier 4) set cluster.tier-max-files to 25 5) heated all 100 files from the fuse mount by writing data 6) only 50 files were promoted Fix works as expected. Help for cluster.tier-max-files now shows, Option: cluster.tier-max-files Default Value: 50000 Description: The maximum number of files that may be migrated in any direction in a given cycle by a single node.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html