Bug 1270527 - add policy mechanism for promotion and demotion
add policy mechanism for promotion and demotion
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
3.7.6
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Dan Lambright
bugs@gluster.org
:
Depends On: 1257911 1271757
Blocks: 1260923 glusterfs-3.7.6
  Show dependency treegraph
 
Reported: 2015-10-10 13:38 EDT by Dan Lambright
Modified: 2015-11-17 00:59 EST (History)
2 users (show)

See Also:
Fixed In Version: glusterfs-3.7.6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1257911
Environment:
Last Closed: 2015-11-17 00:59:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Lambright 2015-10-10 13:38:59 EDT
+++ This bug was initially created as a clone of Bug #1257911 +++

Current mechanism for driving promotion/demotion is too rigorous. Fils are demoted automatically and the hot tier will be underutilized. Need a more flexible system that can be tuned according to customers needs according to various parameters. Need to gracefully handle situations where the hot tier becomes full.

--- Additional comment from Anand Avati on 2015-08-28 07:39:39 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: WIP add watermarks and policy driver) posted (#1) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Anand Avati on 2015-08-28 16:53:20 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: WIP add watermarks and policy driver) posted (#2) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Anand Avati on 2015-09-01 12:31:59 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: WIP add watermarks and policy driver) posted (#3) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-03 17:30:17 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: WIP add watermarks and policy driver) posted (#4) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-08 17:01:37 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: WIP add watermarks and policy driver) posted (#5) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-10 17:56:30 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: add watermarks and policy driver) posted (#6) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-11 20:37:14 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: add watermarks and policy driver) posted (#7) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-18 01:29:54 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: add watermarks and policy driver) posted (#8) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-09 20:46:29 EDT ---

REVIEW: http://review.gluster.org/12039 (cluster/tier: add watermarks and policy driver) posted (#24) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-10 09:00:08 EDT ---

COMMIT: http://review.gluster.org/12039 committed in master by Dan Lambright (dlambrig@redhat.com) 
------
commit 672baab88fb7f32e844cd4be22e0924e4e0e83fc
Author: Dan Lambright <dlambrig@redhat.com>
Date:   Fri Sep 18 00:49:06 2015 -0400

    cluster/tier: add watermarks and policy driver
    
    This fix introduces infrastructure to support different
    policies for promotion and demotion.
    
    Currently the tier feature automatically promotes and demotes
    files periodically based on access. This is good for testing
    but too stringent for most real workloads. It makes it
    difficult to fully utilize a hot tier- data will be demoted
    before it is touched- its unlikely a 100GB hot SSD will have
    all its data touched in a window of time.
    
    A new parameter "mode" allows the user to pick promotion/demotion
    polcies.
    
    The "test mode" will be used for *.t and other general testing.
    This is the current mechanism.
    
    The "cache mode" introduces watermarks. The watermarks
    represent levels of data residing on the hot tier.
    
    "cache mode" policy:
    
    The % the hot tier is full is called P.
    
    Do not promote or demote more than D MB or F files.
    
    A random number [0-100] is called R.
    
    Rules for migration:
    
    if (P < watermark_low) don't demote, always promote.
    
    if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P.
    
    if (P > watermark_hi) always demote, don't promote.
    
    gluster volume set {vol} cluster.watermark-hi %
    gluster volume set {vol} cluster.watermark-low %
    gluster volume set {vol} cluster.tier-max-mb {D}
    gluster volume set {vol} cluster.tier-max-files {F}
    gluster volume set {vol} cluster.tier-mode {test|cache}
    
    Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2
    BUG: 1257911
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    Reviewed-on: http://review.gluster.org/12039
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
Comment 1 Vijay Bellur 2015-10-10 13:39:59 EDT
REVIEW: http://review.gluster.org/12330 (cluster/tier: add watermarks and policy driver) posted (#1) for review on release-3.7 by Dan Lambright (dlambrig@redhat.com)
Comment 2 Vijay Bellur 2015-10-10 17:01:48 EDT
COMMIT: http://review.gluster.org/12330 committed in release-3.7 by Dan Lambright (dlambrig@redhat.com) 
------
commit 98fa496c211dc0da7bccb68fc57f97d835e56c28
Author: Dan Lambright <dlambrig@redhat.com>
Date:   Fri Sep 18 00:49:06 2015 -0400

    cluster/tier: add watermarks and policy driver
    
    Backport fix 12039
    
    This fix introduces infrastructure to support different
    policies for promotion and demotion.
    
    Currently the tier feature automatically promotes and demotes
    files periodically based on access. This is good for testing
    but too stringent for most real workloads. It makes it
    difficult to fully utilize a hot tier- data will be demoted
    before it is touched- its unlikely a 100GB hot SSD will have
    all its data touched in a window of time.
    
    A new parameter "mode" allows the user to pick promotion/demotion
    polcies.
    
    The "test mode" will be used for *.t and other general testing.
    This is the current mechanism.
    
    The "cache mode" introduces watermarks. The watermarks
    represent levels of data residing on the hot tier.
    
    "cache mode" policy:
    
    The % the hot tier is full is called P.
    
    Do not promote or demote more than D MB or F files.
    
    A random number [0-100] is called R.
    
    Rules for migration:
    
    if (P < watermark_low) don't demote, always promote.
    
    if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P.
    
    if (P > watermark_hi) always demote, don't promote.
    
    gluster volume set {vol} cluster.watermark-hi %
    gluster volume set {vol} cluster.watermark-low %
    gluster volume set {vol} cluster.tier-max-mb {D}
    gluster volume set {vol} cluster.tier-max-files {F}
    gluster volume set {vol} cluster.tier-mode {test|cache}
    
    > Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2
    > BUG: 1257911
    > Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    > Reviewed-on: http://review.gluster.org/12039
    > Tested-by: Gluster Build System <jenkins@build.gluster.com>
    > Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    
    Conflicts:
    	xlators/cluster/dht/src/dht-rebalance.c
    	xlators/cluster/dht/src/tier.c
    
    Change-Id: Ibfe6b89563ceab98708325cf5d5ab0997c64816c
    BUG: 1270527
    Reviewed-on: http://review.gluster.org/12330
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Dan Lambright <dlambrig@redhat.com>
    Tested-by: Dan Lambright <dlambrig@redhat.com>
Comment 3 Raghavendra Talur 2015-11-17 00:59:41 EST
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.6, please open a new bug report.

glusterfs-3.7.6 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-November/024359.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.