Bug 1274100 - need a way to pause/stop tiering to take snapshot
need a way to pause/stop tiering to take snapshot
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
3.7.6
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: bugs@gluster.org
bugs@gluster.org
:
Depends On: 1267950 1274334
Blocks: 1260923
  Show dependency treegraph
 
Reported: 2015-10-21 19:02 EDT by Dan Lambright
Modified: 2016-04-19 03:47 EDT (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.7.7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1267950
Environment:
Last Closed: 2016-02-15 03:38:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Lambright 2015-10-21 19:02:38 EDT
+++ This bug was initially created as a clone of Bug #1267950 +++

We need a way to pause tiering to take snapshot. A file cannot be in the middle of promotion/demotion when a snapshot is taken. If a file is in the middle of migration, abort the migration and rollback any in-progress work. Implement a CLI for this and to resume migration.

--- Additional comment from Vijay Bellur on 2015-10-13 21:24:12 EDT ---

REVIEW: http://review.gluster.org/12304 (cluster/tier: WIP add pause tier feature (required for snapshots)) posted (#3) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-14 23:11:23 EDT ---

REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#4) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-16 11:36:14 EDT ---

REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#5) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-19 14:24:17 EDT ---

REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#6) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-21 10:27:52 EDT ---

REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#7) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-21 12:20:15 EDT ---

REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#8) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-10-21 18:44:40 EDT ---

COMMIT: http://review.gluster.org/12304 committed in master by Dan Lambright (dlambrig@redhat.com) 
------
commit 36974c36fa4231df3f0e9428a9da6d1aa33348ab
Author: Dan Lambright <dlambrig@redhat.com>
Date:   Mon Oct 5 19:52:02 2015 +0000

    cluster/tier: add pause tier for snapshots
    
    Snaps of tiered volumes cannot handle files undergoing migration.
    We implement a helper mechanism to "pause" migration. Any files
    undergoing migration are aborted. Clean up is done to remove
    sticky bits and data at the destination. Migration is restarted
    after snap completes.
    
    For testing an internal switch is added. It is not exposed externally.
    
    gluster volume set vol1 tier-pause [true|false]
    
    Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799
    BUG: 1267950
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    Reviewed-on: http://review.gluster.org/12304
    Reviewed-by: N Balachandran <nbalacha@redhat.com>
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
Comment 1 Vijay Bellur 2015-12-24 10:59:01 EST
REVIEW: http://review.gluster.org/13078 (cluster/tier: do not block in synctask created from pause tier) posted (#1) for review on release-3.7 by Dan Lambright (dlambrig@redhat.com)
Comment 2 Vijay Bellur 2015-12-25 08:13:18 EST
COMMIT: http://review.gluster.org/13078 committed in release-3.7 by Dan Lambright (dlambrig@redhat.com) 
------
commit 4ec0e36fff703d156c52bada758b97eff86ce61c
Author: Dan Lambright <dlambrig@redhat.com>
Date:   Wed Dec 16 15:28:57 2015 -0500

    cluster/tier: do not block in synctask created from pause tier
    
    We had run sleep() in the pause tier callback. Blocking within
    a synctask is dangerous. The sleep() call does not inform
    the synctask scheduler that a thread is no longer running.
    It therefore believes it is running. If a second synctask already
    exists, it may not be able to run. This occurs if the thread
    limit in the pool has been reached.
    
    Note the pool size only grows when a synctask is created, not
    when it is moved from wait state to run state, as is the case
    when an FOP completes. When the tier is paused during migration,
    synctasks already exist waiting for responses to FOPs to the
    server with high probability.
    
    The fix is to yield() in the RPC callback, which will place
    the synctask into the wait queue and free up a thread for the
    FOP callback. A timer wakes the callback after sufficient
    time has elapsed for the pause to occur.
    
    This is a backport of 12987.
    
    > Change-Id: I6a947ee04c6e5649946cb6d8207ba17263a67fc6
    > BUG: 1267950
    > Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    > Reviewed-on: http://review.gluster.org/12987
    > Tested-by: Gluster Build System <jenkins@build.gluster.com>
    > Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    
    Change-Id: I9bb7564d57d59abb98e89c8d54c8b1ff4a5fc3f3
    BUG: 1274100
    Reviewed-on: http://review.gluster.org/13078
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Dan Lambright <dlambrig@redhat.com>
    Tested-by: Dan Lambright <dlambrig@redhat.com>
Comment 3 Kaushal 2016-04-19 03:47:08 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.7, please open a new bug report.

glusterfs-3.7.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-February/025292.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.