Bug 1291152 - [tiering]: cluster.tier-max-files option in tiering is not honored
[tiering]: cluster.tier-max-files option in tiering is not honored
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity medium
: ---
: RHGS 3.1.2
Assigned To: Bug Updates Notification Mailing List
krishnaram Karthick
: ZStream
Depends On:
Blocks: 1292671 1292945
  Show dependency treegraph
 
Reported: 2015-12-14 02:05 EST by krishnaram Karthick
Modified: 2016-09-17 11:43 EDT (History)
6 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-13
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1292671 (view as bug list)
Environment:
Last Closed: 2016-03-01 01:02:46 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0193 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 update 2 2016-03-01 05:20:36 EST

  None (edit)
Description krishnaram Karthick 2015-12-14 02:05:29 EST
Description of problem:

Files are not promoted based on the values set in 'cluster.tier-max-files'. When cluster.tier-max-files was set to '2' and 5 files were heated, all 5 files were promoted instead of 2 files.

Version-Release number of selected component (if applicable):
glusterfs-3.7.5-11.el7rhgs.x86_64

How reproducible:
100%

Steps to Reproduce:
1. set 'cluster.tier-max-files' to 2
2. Heat 5 files from cold tier

Actual results:
All 5 files are promoted

Expected results:
only 2 files have to be promoted

Additional info:
sosreports will be attached shortly. Files that were used for this test are /mnt/tier-vol-01/file-99{1..8}
Comment 2 krishnaram Karthick 2015-12-14 02:19:35 EST
sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291152/
Comment 3 krishnaram Karthick 2015-12-14 08:07:55 EST
o/p of gluster vol info

# gluster vol info
 
Volume Name: tier-vol-01
Type: Tier
Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60
Status: Started
Number of Bricks: 16
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.42.10:/rhs/brick4/leg1
Brick2: 10.70.42.177:/rhs/brick4/leg1
Brick3: 10.70.43.19:/rhs/brick4/leg1
Brick4: 10.70.42.47:/rhs/brick4/leg1
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick5: 10.70.42.47:/rhs/brick1/leg1
Brick6: 10.70.43.19:/rhs/brick1/leg1
Brick7: 10.70.42.177:/rhs/brick1/leg1
Brick8: 10.70.42.10:/rhs/brick1/leg1
Brick9: 10.70.43.140:/rhs/brick1/leg1
Brick10: 10.70.42.87:/rhs/brick1/leg1
Brick11: 10.70.42.228:/rhs/brick1/leg1
Brick12: 10.70.42.183:/rhs/brick1/leg1
Brick13: 10.70.42.47:/rhs/brick2/leg1
Brick14: 10.70.43.19:/rhs/brick2/leg1
Brick15: 10.70.42.177:/rhs/brick2/leg1
Brick16: 10.70.42.10:/rhs/brick2/leg1
Options Reconfigured:
cluster.tier-max-files: 2
cluster.tier-promote-frequency: 120
cluster.tier-mode: cache
cluster.tier-demote-frequency: 120
features.record-counters: on
cluster.watermark-hi: 10
cluster.watermark-low: 5
features.ctr-enabled: on
performance.readdir-ahead: on
Comment 4 Dan Lambright 2015-12-14 14:58:34 EST
You are running 4 nodes on the hot tier. The limit tier-max-files is effective per node. Because files are distributed across the 4 nodes, the scenario you describe is legal as each node will individually not cross the limit even though the sum of all the nodes will. Suggest to retry with just a single node on the hot tier.
Comment 5 krishnaram Karthick 2015-12-15 04:53:40 EST
When hot tier is made of 1 node and cluster.tier-max-files is set to 1, more than 1 file gets promoted in a single cycle.

When 9 files were heated 
 - On my first attempt 2 files were promoted
 - second attempt 2 files were promoted
 - third attempt 3 files were promoted
 - Fourth attempt 4 files were promoted

As a whole, the behavior is not consistent, but it is pretty sure that the value set for 'cluster.tier-max-files' isn't honored.

# gluster vol info

Volume Name: tier-vol-01
Type: Tier
Volume ID: 78d1fc18-5d0a-452d-ab26-b78c15655c60
Status: Started
Number of Bricks: 13
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.70.42.47:/rhs/brick4/leg2
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick2: 10.70.42.47:/rhs/brick1/leg1
Brick3: 10.70.43.19:/rhs/brick1/leg1
Brick4: 10.70.42.177:/rhs/brick1/leg1
Brick5: 10.70.42.10:/rhs/brick1/leg1
Brick6: 10.70.43.140:/rhs/brick1/leg1
Brick7: 10.70.42.87:/rhs/brick1/leg1
Brick8: 10.70.42.228:/rhs/brick1/leg1
Brick9: 10.70.42.183:/rhs/brick1/leg1
Brick10: 10.70.42.47:/rhs/brick2/leg1
Brick11: 10.70.43.19:/rhs/brick2/leg1
Brick12: 10.70.42.177:/rhs/brick2/leg1
Brick13: 10.70.42.10:/rhs/brick2/leg1
Options Reconfigured:
cluster.tier-max-files: 1
cluster.tier-mode: cache
features.ctr-enabled: on
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 120
features.record-counters: on
cluster.watermark-hi: 60
cluster.watermark-low: 30
performance.readdir-ahead: on

Moreover, as per the definition of 'cluster.tier-max-files' in help, we don't define that this setting is based on number of nodes. It is easy to assume that this setting applies as a whole to a volume. We'll have to modify our help section to reflect the actual behavior.

Option: cluster.tier-max-files
Default Value: 5000
Description: The maximum number of files that may be migrated in any direction in a given cycle.

sosreport for today's attempt to reproduce will be attached shortly.
Comment 6 krishnaram Karthick 2015-12-15 04:58:46 EST
Files that got promoted when files-{981..999} were heated up are pasted below.

[root@dhcp42-47 tmp]# ll -lh /rhs/brick4/leg2/dd/
total 1.1G
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-981
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-983
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-985
-rw-r--r--. 2 root root 102M Dec 15 15:16 file-987
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-992
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-993
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-994
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-995
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-996
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-997
-rw-r--r--. 2 root root 102M Dec 15 15:11 file-999
Comment 7 krishnaram Karthick 2015-12-15 05:06:13 EST
sosreports are available here
Comment 8 Dan Lambright 2015-12-15 16:25:47 EST
You said you tried with one node, but your volume info shows 10.70.42.47, 10.70.43.19, etc. ; Each IP address would be different node. 

Is "tier-vol-01" the volume you tried? If not, can you paste what was? 

I do not see any link to sos reports in comment 7, can you post that as well?

We may be allowing migration of max-migrate-files+1 rather than max-migrate_files. Your comment in the help message w.r.t documentation is valid and could be fixed using this bug as a vehicle.
Comment 9 krishnaram Karthick 2015-12-16 08:36:29 EST
The volume under test was tier-vol-01. Although the test system has 8 nodes, only one brick out of all 8 nodes is used as hot tier. i.e., Brick1: 10.70.42.47:/rhs/brick4/leg2. Rest of the bricks seen in vol info o/p are cold tier.

sosreports --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1291152/reproduced_on_15dec/
Comment 10 Dan Lambright 2015-12-16 15:53:23 EST
The limit is on the node in the migration source, not the destination. You are doing promotions, so up to 6 nodes may be source nodes as your cold tier in a given cycle.
Comment 11 Vivek Agarwal 2015-12-17 04:11:09 EST
http://review.gluster.org/12984
Comment 14 krishnaram Karthick 2015-12-24 04:27:42 EST
Verified the fix in build glusterfs-server-3.7.5-13.el7rhgs.x86_64

Steps followed to verify:

1) configured a 1 brick gluster volume
2) created 100 files
3) Attached a 1 brick hot tier
4) set cluster.tier-max-files to 50
5) heated all 100 files from the fuse mount by writing data
6) only 50 files were promoted

Repeated the test with 2 brick distributed cold tier and 1 brick hot tier.

steps:

1) configured a 2 brick gluster volume
2) created 100 files
3) Attached a 1 brick hot tier
4) set cluster.tier-max-files to 25
5) heated all 100 files from the fuse mount by writing data
6) only 50 files were promoted

Fix works as expected.

Help for cluster.tier-max-files now shows,

Option: cluster.tier-max-files
Default Value: 50000
Description: The maximum number of files that may be migrated in any direction in a given cycle by a single node.
Comment 16 errata-xmlrpc 2016-03-01 01:02:46 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.