Bug 1220041 - timer wheel and throttling in bitrot
Summary: timer wheel and throttling in bitrot
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: bitrot
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-09 12:37 UTC by Raghavendra Bhat
Modified: 2015-05-14 17:47 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-05-14 17:29:41 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Raghavendra Bhat 2015-05-09 12:37:29 UTC
Description of problem:

Backport the timer wheel related patches along with throttling changes in bitrot xlator from master branch. Also needed is the changes to manage fds.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Anand Avati 2015-05-09 13:12:39 UTC
REVIEW: http://review.gluster.org/10718 (features/bitrot: Follow xattr naming conventions) posted (#1) for review on release-3.7 by Raghavendra Bhat (raghavendra)

Comment 2 Anand Avati 2015-05-09 13:13:02 UTC
REVIEW: http://review.gluster.org/10719 (features/bit-rot: Token Bucket based throttling) posted (#1) for review on release-3.7 by Raghavendra Bhat (raghavendra)

Comment 3 Anand Avati 2015-05-09 13:13:13 UTC
REVIEW: http://review.gluster.org/10720 (features/bitrot: Throttle filesystem scrubber) posted (#1) for review on release-3.7 by Raghavendra Bhat (raghavendra)

Comment 4 Anand Avati 2015-05-09 13:13:15 UTC
REVIEW: http://review.gluster.org/10721 (features/bitrot: Scrubber pause/resume) posted (#1) for review on release-3.7 by Raghavendra Bhat (raghavendra)

Comment 5 Anand Avati 2015-05-09 13:13:26 UTC
REVIEW: http://review.gluster.org/10722 (features/bit-rot-stub: versioning of objects in write/truncate fop instead of open) posted (#1) for review on release-3.7 by Raghavendra Bhat (raghavendra)

Comment 6 Anand Avati 2015-05-09 17:30:42 UTC
REVIEW: http://review.gluster.org/10714 (bitrot/scrub: fix induced throttling in syncop_ftw_throttle()) posted (#2) for review on release-3.7 by Vijay Bellur (vbellur)

Comment 7 Anand Avati 2015-05-09 17:31:09 UTC
REVIEW: http://review.gluster.org/10715 (bitrot: Scrubber log should report 'bad' file detection as ALERT in log) posted (#2) for review on release-3.7 by Vijay Bellur (vbellur)

Comment 8 Anand Avati 2015-05-09 21:44:09 UTC
REVIEW: http://review.gluster.org/10716 (core: Global timer-wheel) posted (#2) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 9 Anand Avati 2015-05-09 21:44:50 UTC
REVIEW: http://review.gluster.org/10718 (features/bitrot: Follow xattr naming conventions) posted (#2) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 10 Anand Avati 2015-05-09 21:44:52 UTC
REVIEW: http://review.gluster.org/10719 (features/bit-rot: Token Bucket based throttling) posted (#2) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 11 Anand Avati 2015-05-09 21:45:05 UTC
REVIEW: http://review.gluster.org/10714 (bitrot/scrub: fix induced throttling in syncop_ftw_throttle()) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 12 Anand Avati 2015-05-09 21:45:16 UTC
REVIEW: http://review.gluster.org/10715 (bitrot: Scrubber log should report 'bad' file detection as ALERT in log) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 13 Anand Avati 2015-05-09 21:45:19 UTC
REVIEW: http://review.gluster.org/10721 (features/bitrot: Scrubber pause/resume) posted (#2) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 14 Anand Avati 2015-05-09 21:45:21 UTC
REVIEW: http://review.gluster.org/10720 (features/bitrot: Throttle filesystem scrubber) posted (#2) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 15 Anand Avati 2015-05-10 03:49:29 UTC
COMMIT: http://review.gluster.org/10715 committed in release-3.7 by Vijay Bellur (vbellur) 
------
commit 8f267cc87b364970db3444620bb53dc2f50ca13b
Author: Gaurav Kumar Garg <ggarg>
Date:   Tue Apr 14 00:48:24 2015 +0530

    bitrot: Scrubber log should report 'bad' file detection as ALERT in log
    
    If scrubber detect any bad object by mismatching of checksum of scrubber
    and signer then log messages shold come as a Alert instead of warning.
    
    > Change-Id: I075d80700cbe6182e525a04419a80ab18419ff91
    > BUG: 1210687
    > Signed-off-by: Gaurav Kumar Garg <ggarg>
    > Reviewed-on: http://review.gluster.org/10226
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Raghavendra Bhat <raghavendra>
    
    Change-Id: I7c733c82aed5a00c74e60dc7baca0aa9acf26fad
    BUG: 1220041
    Reviewed-on: http://review.gluster.org/10715
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System
    Reviewed-by: Vijay Bellur <vbellur>

Comment 16 Anand Avati 2015-05-10 04:59:06 UTC
REVIEW: http://review.gluster.org/10722 (features/bit-rot-stub: versioning of objects in write/truncate fop instead of open) posted (#2) for review on release-3.7 by Vijay Bellur (vbellur)

Comment 17 Anand Avati 2015-05-10 05:05:00 UTC
REVIEW: http://review.gluster.org/10716 (core: Global timer-wheel) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 18 Anand Avati 2015-05-10 05:05:37 UTC
REVIEW: http://review.gluster.org/10718 (features/bitrot: Follow xattr naming conventions) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 19 Anand Avati 2015-05-10 05:06:00 UTC
REVIEW: http://review.gluster.org/10719 (features/bit-rot: Token Bucket based throttling) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 20 Anand Avati 2015-05-10 05:06:14 UTC
REVIEW: http://review.gluster.org/10721 (features/bitrot: Scrubber pause/resume) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 21 Anand Avati 2015-05-10 05:06:16 UTC
REVIEW: http://review.gluster.org/10720 (features/bitrot: Throttle filesystem scrubber) posted (#3) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 22 Anand Avati 2015-05-10 09:18:58 UTC
REVIEW: http://review.gluster.org/10716 (core: Global timer-wheel) posted (#4) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 23 Anand Avati 2015-05-10 09:19:00 UTC
REVIEW: http://review.gluster.org/10717 (features/bitrot: Use global timer wheel) posted (#4) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 24 Anand Avati 2015-05-10 09:19:02 UTC
REVIEW: http://review.gluster.org/10718 (features/bitrot: Follow xattr naming conventions) posted (#4) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 25 Anand Avati 2015-05-10 09:19:04 UTC
REVIEW: http://review.gluster.org/10719 (features/bit-rot: Token Bucket based throttling) posted (#4) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 26 Anand Avati 2015-05-10 09:19:06 UTC
REVIEW: http://review.gluster.org/10721 (features/bitrot: Scrubber pause/resume) posted (#4) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 27 Anand Avati 2015-05-10 09:19:11 UTC
REVIEW: http://review.gluster.org/10720 (features/bitrot: Throttle filesystem scrubber) posted (#4) for review on release-3.7 by Gaurav Kumar Garg (ggarg)

Comment 28 Anand Avati 2015-05-10 12:27:43 UTC
COMMIT: http://review.gluster.org/10716 committed in release-3.7 by Niels de Vos (ndevos) 
------
commit 1a217b2a0295ca4d9068ee5c17d6a4374cc5f8fc
Author: Venky Shankar <vshankar>
Date:   Fri Apr 24 10:10:35 2015 +0530

    core: Global timer-wheel
    
    Instantiate a process wide global instance of the timer wheel
    data structure. Spawning glusterfs* process with option arg
    "--global-timer-wheel" instantiates a global instance of
    timer-wheel under global context (->ctx).
    
    Translators can make use of this process wide instance [via a
    call to glusterfs_global_timer_wheel()] instead of maintaining
    an instance of their own and possibly consuming more memory.
    Linux kernel too has a single instance of timer wheel where
    subsystems such as IO, networking, etc.. make use of.
    
    Bitrot daemon would be early consumers of this: bitrot translator
    instances for multiple volumes would track objects belonging to
    their respective bricks in this global expiry tracking data
    structure. This is also a first step to move GlusterFS timer
    mechanism to use timer-wheel.
    
    > Change-Id: Ie882df607e07acaced846ea269ebf1ece306d6ae
    > BUG: 1170075
    > Signed-off-by: Venky Shankar <vshankar>
    > Reviewed-on: http://review.gluster.org/10380
    > Tested-by: NetBSD Build System
    > Reviewed-by: Vijay Bellur <vbellur>
    > Tested-by: Gluster Build System <jenkins.com>
    
    Change-Id: I35c840daa9996a059699f8ea5af54c76ede7e09c
    Signed-off-by: Venky Shankar <vshankar>
    Signed-off-by: Gaurav Kumar Garg <ggarg>
    BUG: 1220041
    Reviewed-on: http://review.gluster.org/10716
    Tested-by: Gluster Build System <jenkins.com>

Comment 29 Anand Avati 2015-05-10 12:28:25 UTC
COMMIT: http://review.gluster.org/10717 committed in release-3.7 by Niels de Vos (ndevos) 
------
commit e043bd6b3b80aa17a9853f223bf86e6311060e01
Author: Venky Shankar <vshankar>
Date:   Sun Apr 26 16:25:08 2015 +0530

    features/bitrot: Use global timer wheel
    
    > Change-Id: I761927ea263b4144b851881f25791fda5b794f59
    > BUG: 1170075
    > Signed-off-by: Venky Shankar <vshankar>
    > Reviewed-on: http://review.gluster.org/10381
    > Tested-by: NetBSD Build System
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Raghavendra Bhat <raghavendra>
    > Reviewed-by: Vijay Bellur <vbellur>
    
    Change-Id: I4aa7c0d8b42b4c8d14a1d810e54c2de4d52b4389
    Signed-off-by: Venky Shankar <vshankar>
    BUG: 1220041
    Reviewed-on: http://review.gluster.org/10717
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Gaurav Kumar Garg <ggarg>

Comment 30 Anand Avati 2015-05-10 12:29:48 UTC
COMMIT: http://review.gluster.org/10720 committed in release-3.7 by Niels de Vos (ndevos) 
------
commit 32865f8650057123a5fcf590c96a1ae3f6d22608
Author: Venky Shankar <vshankar>
Date:   Mon Apr 27 21:34:34 2015 +0530

    features/bitrot: Throttle filesystem scrubber
    
    This patch introduces multithreaded filesystem scrubber based
    on throttling option configured for a particular volume. The
    implementation "logically" breaks scanning and scrubbing with
    the number of scrubber threads auto-configured depending upon
    the throttle configuration. Scanning (crawling) is left single
    threaded (per brick) with entries scrubbed in bulk. On reaching
    this "bulk" watermark, scanner waits until entries are scrubbed.
    Bricks for a particular volume have a set of thread(s) assigned
    for scrubbing, with entries for each brick scrubbed in a round
    robin fashion to avoid scrub "stalls" when a brick (out of N
    bricks) is under active scrubbing.
    
    This mechanism helps us implement "pause/resume" with ease: all
    one need to do is to cleanup scrubber threads and let the main
    scanner thread "wait" untill scrubbing is resumed (where the
    scrubber thread(s) are spawned again), therefore continuing
    where we left off (unless we restart the deamons, where crawl
    initiates from root directory again, but I guess that's OK).
    
    [
        NOTE:
    
        Throttling is optional for the signer daemon, without which
        it runs full throttle. However, passing "-DBR_RATE_LIMIT_SIGNER"
        predefined in CFLAGS enables CPU throttling (during checksum
        calculation) thereby avoiding high CPU usage.
    ]
    
    Subsequent patches would introduce CPU throttling during hash
    calculation for scrubber.
    
    > Change-Id: I5701dd6cd4dff27ca3144ac5e3798a2216b39d4f
    > BUG: 1207020
    > Signed-off-by: Venky Shankar <vshankar>
    > Reviewed-on: http://review.gluster.org/10511
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Vijay Bellur <vbellur>
    
    Change-Id: I5a125b2d0ac7dafd3e278b7fe4c6c9dd07af76dd
    Signed-off-by: Venky Shankar <vshankar>
    BUG: 1220041
    Reviewed-on: http://review.gluster.org/10720
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Gaurav Kumar Garg <ggarg>

Comment 31 Anand Avati 2015-05-10 12:40:27 UTC
REVIEW: http://review.gluster.org/10722 (features/bit-rot-stub: versioning of objects in write/truncate fop instead of open) posted (#3) for review on release-3.7 by Raghavendra Bhat (raghavendra)

Comment 32 Niels de Vos 2015-05-14 17:29:41 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 33 Niels de Vos 2015-05-14 17:36:02 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 34 Niels de Vos 2015-05-14 17:38:23 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 35 Niels de Vos 2015-05-14 17:47:41 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.