Bug 1291152
| Summary: | [tiering]: cluster.tier-max-files option in tiering is not honored | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | krishnaram Karthick <kramdoss> | |
| Component: | tier | Assignee: | Bug Updates Notification Mailing List <rhs-bugs> | |
| Status: | CLOSED ERRATA | QA Contact: | krishnaram Karthick <kramdoss> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.1 | CC: | byarlaga, dlambrig, kramdoss, rhs-bugs, sankarshan, storage-qa-internal | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | RHGS 3.1.2 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.7.5-13 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1292671 (view as bug list) | Environment: | ||
| Last Closed: | 2016-03-01 06:02:46 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1292671, 1292945 | |||
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291152/ o/p of gluster vol info # gluster vol info Volume Name: tier-vol-01 Type: Tier Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60 Status: Started Number of Bricks: 16 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: 10.70.42.10:/rhs/brick4/leg1 Brick2: 10.70.42.177:/rhs/brick4/leg1 Brick3: 10.70.43.19:/rhs/brick4/leg1 Brick4: 10.70.42.47:/rhs/brick4/leg1 Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (4 + 2) = 12 Brick5: 10.70.42.47:/rhs/brick1/leg1 Brick6: 10.70.43.19:/rhs/brick1/leg1 Brick7: 10.70.42.177:/rhs/brick1/leg1 Brick8: 10.70.42.10:/rhs/brick1/leg1 Brick9: 10.70.43.140:/rhs/brick1/leg1 Brick10: 10.70.42.87:/rhs/brick1/leg1 Brick11: 10.70.42.228:/rhs/brick1/leg1 Brick12: 10.70.42.183:/rhs/brick1/leg1 Brick13: 10.70.42.47:/rhs/brick2/leg1 Brick14: 10.70.43.19:/rhs/brick2/leg1 Brick15: 10.70.42.177:/rhs/brick2/leg1 Brick16: 10.70.42.10:/rhs/brick2/leg1 Options Reconfigured: cluster.tier-max-files: 2 cluster.tier-promote-frequency: 120 cluster.tier-mode: cache cluster.tier-demote-frequency: 120 features.record-counters: on cluster.watermark-hi: 10 cluster.watermark-low: 5 features.ctr-enabled: on performance.readdir-ahead: on You are running 4 nodes on the hot tier. The limit tier-max-files is effective per node. Because files are distributed across the 4 nodes, the scenario you describe is legal as each node will individually not cross the limit even though the sum of all the nodes will. Suggest to retry with just a single node on the hot tier. When hot tier is made of 1 node and cluster.tier-max-files is set to 1, more than 1 file gets promoted in a single cycle. When 9 files were heated - On my first attempt 2 files were promoted - second attempt 2 files were promoted - third attempt 3 files were promoted - Fourth attempt 4 files were promoted As a whole, the behavior is not consistent, but it is pretty sure that the value set for 'cluster.tier-max-files' isn't honored. # gluster vol info Volume Name: tier-vol-01 Type: Tier Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60 Status: Started Number of Bricks: 13 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.70.42.47:/rhs/brick4/leg2 Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (4 + 2) = 12 Brick2: 10.70.42.47:/rhs/brick1/leg1 Brick3: 10.70.43.19:/rhs/brick1/leg1 Brick4: 10.70.42.177:/rhs/brick1/leg1 Brick5: 10.70.42.10:/rhs/brick1/leg1 Brick6: 10.70.43.140:/rhs/brick1/leg1 Brick7: 10.70.42.87:/rhs/brick1/leg1 Brick8: 10.70.42.228:/rhs/brick1/leg1 Brick9: 10.70.42.183:/rhs/brick1/leg1 Brick10: 10.70.42.47:/rhs/brick2/leg1 Brick11: 10.70.43.19:/rhs/brick2/leg1 Brick12: 10.70.42.177:/rhs/brick2/leg1 Brick13: 10.70.42.10:/rhs/brick2/leg1 Options Reconfigured: cluster.tier-max-files: 1 cluster.tier-mode: cache features.ctr-enabled: on cluster.tier-promote-frequency: 120 cluster.tier-demote-frequency: 120 features.record-counters: on cluster.watermark-hi: 60 cluster.watermark-low: 30 performance.readdir-ahead: on Moreover, as per the definition of 'cluster.tier-max-files' in help, we don't define that this setting is based on number of nodes. It is easy to assume that this setting applies as a whole to a volume. We'll have to modify our help section to reflect the actual behavior. Option: cluster.tier-max-files Default Value: 5000 Description: The maximum number of files that may be migrated in any direction in a given cycle. sosreport for today's attempt to reproduce will be attached shortly. Files that got promoted when files-{981..999} were heated up are pasted below.
[root@dhcp42-47 tmp]# ll -lh /rhs/brick4/leg2/dd/
total 1.1G
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-981
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-983
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-985
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-987
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-992
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-993
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-994
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-995
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-996
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-997
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-999
sosreports are available here You said you tried with one node, but your volume info shows 10.70.42.47, 10.70.43.19, etc. ; Each IP address would be different node. Is "tier-vol-01" the volume you tried? If not, can you paste what was? I do not see any link to sos reports in comment 7, can you post that as well? We may be allowing migration of max-migrate-files+1 rather than max-migrate_files. Your comment in the help message w.r.t documentation is valid and could be fixed using this bug as a vehicle. The volume under test was tier-vol-01. Although the test system has 8 nodes, only one brick out of all 8 nodes is used as hot tier. i.e., Brick1: 10.70.42.47:/rhs/brick4/leg2. Rest of the bricks seen in vol info o/p are cold tier. sosreports --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291152/reproduced_on_15dec/ The limit is on the node in the migration source, not the destination. You are doing promotions, so up to 6 nodes may be source nodes as your cold tier in a given cycle. Verified the fix in build glusterfs-server-3.7.5-13.el7rhgs.x86_64 Steps followed to verify: 1) configured a 1 brick gluster volume 2) created 100 files 3) Attached a 1 brick hot tier 4) set cluster.tier-max-files to 50 5) heated all 100 files from the fuse mount by writing data 6) only 50 files were promoted Repeated the test with 2 brick distributed cold tier and 1 brick hot tier. steps: 1) configured a 2 brick gluster volume 2) created 100 files 3) Attached a 1 brick hot tier 4) set cluster.tier-max-files to 25 5) heated all 100 files from the fuse mount by writing data 6) only 50 files were promoted Fix works as expected. Help for cluster.tier-max-files now shows, Option: cluster.tier-max-files Default Value: 50000 Description: The maximum number of files that may be migrated in any direction in a given cycle by a single node. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html |
Description of problem: Files are not promoted based on the values set in 'cluster.tier-max-files'. When cluster.tier-max-files was set to '2' and 5 files were heated, all 5 files were promoted instead of 2 files. Version-Release number of selected component (if applicable): glusterfs-3.7.5-11.el7rhgs.x86_64 How reproducible: 100% Steps to Reproduce: 1. set 'cluster.tier-max-files' to 2 2. Heat 5 files from cold tier Actual results: All 5 files are promoted Expected results: only 2 files have to be promoted Additional info: sosreports will be attached shortly. Files that were used for this test are /mnt/tier-vol-01/file-99{1..8}