Bug 1267950 - need a way to pause/stop tiering to take snapshot
need a way to pause/stop tiering to take snapshot
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Dan Lambright
bugs@gluster.org
:
Depends On:
Blocks: 1274100 1274101 1274334
  Show dependency treegraph
 
Reported: 2015-10-01 08:43 EDT by Dan Lambright
Modified: 2016-06-16 09:39 EDT (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1274100 1274101 1274334 (view as bug list)
Environment:
Last Closed: 2016-06-16 09:39:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Lambright 2015-10-01 08:43:38 EDT
We need a way to pause tiering to take snapshot. A file cannot be in the middle of promotion/demotion when a snapshot is taken. If a file is in the middle of migration, abort the migration and rollback any in-progress work. Implement a CLI for this and to resume migration.
Comment 1 Vijay Bellur 2015-10-13 21:24:12 EDT
REVIEW: http://review.gluster.org/12304 (cluster/tier: WIP add pause tier feature (required for snapshots)) posted (#3) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 2 Vijay Bellur 2015-10-14 23:11:23 EDT
REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#4) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 3 Vijay Bellur 2015-10-16 11:36:14 EDT
REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#5) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 4 Vijay Bellur 2015-10-19 14:24:17 EDT
REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#6) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 5 Vijay Bellur 2015-10-21 10:27:52 EDT
REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#7) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 6 Vijay Bellur 2015-10-21 12:20:15 EDT
REVIEW: http://review.gluster.org/12304 (cluster/tier: add pause tier for snapshots) posted (#8) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 7 Vijay Bellur 2015-10-21 18:44:40 EDT
COMMIT: http://review.gluster.org/12304 committed in master by Dan Lambright (dlambrig@redhat.com) 
------
commit 36974c36fa4231df3f0e9428a9da6d1aa33348ab
Author: Dan Lambright <dlambrig@redhat.com>
Date:   Mon Oct 5 19:52:02 2015 +0000

    cluster/tier: add pause tier for snapshots
    
    Snaps of tiered volumes cannot handle files undergoing migration.
    We implement a helper mechanism to "pause" migration. Any files
    undergoing migration are aborted. Clean up is done to remove
    sticky bits and data at the destination. Migration is restarted
    after snap completes.
    
    For testing an internal switch is added. It is not exposed externally.
    
    gluster volume set vol1 tier-pause [true|false]
    
    Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799
    BUG: 1267950
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    Reviewed-on: http://review.gluster.org/12304
    Reviewed-by: N Balachandran <nbalacha@redhat.com>
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
Comment 8 Vijay Bellur 2015-12-10 15:59:16 EST
REVIEW: http://review.gluster.org/12934 (cluster/tier: fix pause tier) posted (#3) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 9 Vijay Bellur 2015-12-11 14:37:23 EST
REVIEW: http://review.gluster.org/12934 (cluster/tier: fix pause tier) posted (#4) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 10 Vijay Bellur 2015-12-13 18:16:13 EST
REVIEW: http://review.gluster.org/12934 (cluster/tier: fix pause tier) posted (#5) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 11 Vijay Bellur 2015-12-16 15:47:50 EST
REVIEW: http://review.gluster.org/12987 (cluster/tier: do not block in synctask created from pause tier) posted (#1) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 12 Dan Lambright 2015-12-16 15:58:32 EST
12987 shall supersede 12934 per analysis of code, stack [1], discussions, and ultimately testing the correction over the reproduced original problem.

[1]
http://ur1.ca/ocgrq
Comment 13 Vijay Bellur 2015-12-18 15:51:08 EST
REVIEW: http://review.gluster.org/12987 (cluster/tier: do not block in synctask created from pause tier) posted (#2) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 14 Vijay Bellur 2015-12-20 08:06:35 EST
REVIEW: http://review.gluster.org/12987 (cluster/tier: do not block in synctask created from pause tier) posted (#3) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 15 Vijay Bellur 2015-12-21 08:23:23 EST
COMMIT: http://review.gluster.org/12987 committed in master by Dan Lambright (dlambrig@redhat.com) 
------
commit 3a094f1de03b3da8cdff650c14e46aab87e1905b
Author: Dan Lambright <dlambrig@redhat.com>
Date:   Wed Dec 16 15:28:57 2015 -0500

    cluster/tier: do not block in synctask created from pause tier
    
    We had run sleep() in the pause tier callback. Blocking within
    a synctask is dangerous. The sleep() call does not inform
    the synctask scheduler that a thread is no longer running.
    It therefore believes it is running. If a second synctask already
    exists, it may not be able to run. This occurs if the thread
    limit in the pool has been reached.
    
    Note the pool size only grows when a synctask is created, not
    when it is moved from wait state to run state, as is the case
    when an FOP completes. When the tier is paused during migration,
    synctasks already exist waiting for responses to FOPs to the
    server with high probability.
    
    The fix is to yield() in the RPC callback, which will place
    the synctask into the wait queue and free up a thread for the
    FOP callback. A timer wakes the callback after sufficient
    time has elapsed for the pause to occur.
    
    Change-Id: I6a947ee04c6e5649946cb6d8207ba17263a67fc6
    BUG: 1267950
    Signed-off-by: Dan Lambright <dlambrig@redhat.com>
    Reviewed-on: http://review.gluster.org/12987
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
Comment 16 Niels de Vos 2016-06-16 09:39:10 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.