Bug 1403192

Summary:	Files remain unhealed forever if shd is disabled and re-enabled while healing is in progress.
Product:	[Community] GlusterFS	Reporter:	Ravishankar N <ravishankar>
Component:	replicate	Assignee:	Ravishankar N <ravishankar>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.8	CC:	bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.8.7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1402841	Environment:
Last Closed:	2016-12-14 07:13:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1402841
Bug Blocks:	1403120, 1403187

Description Ravishankar N 2016-12-09 11:52:43 UTC

+++ This bug was initially created as a clone of Bug #1402841 +++

Description of problem:

1. Create a 1x2 replica vol using a 2 node cluster.
2. Fuse mount the vol and create 2000 files
3. Bring one brick down, write to those files, leading to 2000 pending data heals.
4. Bring back the brick and launch index heal
5. The shd log on the source brick prints completed heals for the the processed files.
6. Before the heal completes, do a `gluster vol set volname self-heal-daemon off`
7. The heal stops as expected.
8. Re-enable the shd: `gluster vol set volname self-heal-daemon on`
9. Observe the shd log, we don't see any files getting healed.
10. Launching index heal manually also has no effect.

The only workaround is to restart shd with a `volume start force`.

--- Additional comment from Worker Ant on 2016-12-08 07:55:33 EST ---

REVIEW: http://review.gluster.org/16073 (syncop: fix conditional wait bug in parallel dir scan) posted (#1) for review on master by Ravishankar N (ravishankar)

--- Additional comment from Worker Ant on 2016-12-09 00:27:13 EST ---

REVIEW: http://review.gluster.org/16073 (syncop: fix conditional wait bug in parallel dir scan) posted (#2) for review on master by Ravishankar N (ravishankar)

--- Additional comment from Worker Ant on 2016-12-09 05:24:25 EST ---

COMMIT: http://review.gluster.org/16073 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 2d012c4558046afd6adb3992ff88f937c5f835e4
Author: Ravishankar N <ravishankar>
Date:   Fri Dec 9 09:50:43 2016 +0530

    syncop: fix conditional wait bug in parallel dir scan
    
    Problem:
    The issue as seen by the user is detailed in the BZ but what is
    happening is if the no. of items in the wait queue == max-qlen,
    syncop_mt_dir_scan() does a pthread_cond_wait until the launched
    synctask workers dequeue the queue. But if for some reason the worker
    fails, the queue is never emptied due to which further invocations of
    syncop_mt_dir_scan() are blocked forever.
    
    Fix: Made some changes to _dir_scan_job_fn
    
    - If a worker encounters error while processing an entry, notify the
      readdir loop in syncop_mt_dir_scan() of the error but continue to process
      other entries in the queue, decrementing the qlen as and when we dequeue
      elements, and ending only when the queue is empty.
    
    - If the readdir loop in syncop_mt_dir_scan() gets an error form the
      worker, stop the readdir+queueing of further entries.
    
    Change-Id: I39ce073e01a68c7ff18a0e9227389245a6f75b88
    BUG: 1402841
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/16073
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 1 Worker Ant 2016-12-09 11:53:28 UTC

REVIEW: http://review.gluster.org/16096 (syncop: fix conditional wait bug in parallel dir scan) posted (#1) for review on release-3.8 by Ravishankar N (ravishankar)

Comment 2 Worker Ant 2016-12-12 07:16:50 UTC

COMMIT: http://review.gluster.org/16096 committed in release-3.8 by Pranith Kumar Karampuri (pkarampu) 
------
commit 1d66eb4af160dfa6350410cd6d03e4aa1caf1c53
Author: Ravishankar N <ravishankar>
Date:   Fri Dec 9 09:50:43 2016 +0530

    syncop: fix conditional wait bug in parallel dir scan
    
    Problem:
    The issue as seen by the user is detailed in the BZ but what is
    happening is if the no. of items in the wait queue == max-qlen,
    syncop_mt_dir_scan() does a pthread_cond_wait until the launched
    synctask workers dequeue the queue. But if for some reason the worker
    fails, the queue is never emptied due to which further invocations of
    syncop_mt_dir_scan() are blocked forever.
    
    Fix: Made some changes to _dir_scan_job_fn
    
    - If a worker encounters error while processing an entry, notify the
      readdir loop in syncop_mt_dir_scan() of the error but continue to process
      other entries in the queue, decrementing the qlen as and when we dequeue
      elements, and ending only when the queue is empty.
    
    - If the readdir loop in syncop_mt_dir_scan() gets an error form the
      worker, stop the readdir+queueing of further entries.
    
    > Reviewed-on: http://review.gluster.org/16073
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    (cherry picked from commit 2d012c4558046afd6adb3992ff88f937c5f835e4)
    
    Change-Id: I39ce073e01a68c7ff18a0e9227389245a6f75b88
    BUG: 1403192
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/16096
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 3 Niels de Vos 2016-12-14 07:13:24 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.7, please open a new bug report.

glusterfs-3.8.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/packaging/2016-December/000226.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 4 Worker Ant 2016-12-21 23:21:52 UTC

REVIEW: http://review.gluster.org/16253 (syncop: fix conditional wait bug in parallel dir scan) posted (#1) for review on release-3.8-fb by Kevin Vigor (kvigor)