Bug 1659708

+++ This bug was initially created as a clone of Bug #1471742 +++

Description of problem:
=======================
When there is more than 1 volume in a cluster setup, if we stop one volume, we stop and start even the selfheal deamons on all nodes.
This is completely unncessary and waste of resources.
We are immediately restarting shd because there are still other volumes available. This means we are interfering momentarly in the heal process for any file which was getting healed , and this means the file takes a bit more time to heal.

before stopping the healdeamon as part of volume stop, we must check if there are any other volumes associated with the heal deamon and avoid stopping and regenerating a new shd process when it is not required
Version-Release number of selected component (if applicable):


How reproducible:
========
always


Steps to Reproduce:
1.create 3 volumes all 1x3
2.note down the shd pids
3.now stop v1
4. check the shd pid, it can be seen they are restarted
5. now stop v2, again it can be seen the shd pids are restarted

Actual results:
===========

Expected results:
don't stop shds unless it is the last volume


--- Additional comment from Ravishankar N on 2017-07-18 09:37:10 UTC ---

This behaviour is by design. Not killing and restarting shd with the new graph (containing only active volumes) means we would need to perform graph switch in shd. Unlike other processes (say fuse mount/ brick process etc ) which has a inode table maintained by the top most xlator, in glustershd, each afr maintains its own inode table, so we need to migrate all of them etc. which is not worth it.

Also there is not much loss in state due to a restart because shd only restarts heals of files that were not completed earlier.

--- Additional comment from nchilaka on 2017-07-18 10:08:20 UTC ---

I still see this as an issue, for eg take a case when a 100 gb file was 99gb healed, then in this case we try to scan and heal from beginning.
If the fix is not worth the effort, then we need to move to won't fix or can't fix instead of "not a bug"

--- Additional comment from Pranith Kumar K on 2017-07-18 10:32:14 UTC ---

(In reply to nchilaka from comment #3)
> I still see this as an issue, for eg take a case when a 100 gb file was 99gb
> healed, then in this case we try to scan and heal from beginning.
> If the fix is not worth the effort, then we need to move to won't fix or
> can't fix instead of "not a bug"

yeah I agree, let's move it to WONTFIX. It is a design tradeoff.

--- Additional comment from Ravishankar N on 2017-07-18 10:34:57 UTC ---

(In reply to nchilaka from comment #3)
> I still see this as an issue, for eg take a case when a 100 gb file was 99gb
> healed, then in this case we try to scan and heal from beginning.

For the record, this is something which can be fixed by introducing granular data-self-heal.

--- Additional comment from Pranith Kumar K on 2017-07-18 10:59:49 UTC ---

(In reply to Ravishankar N from comment #5)
> (In reply to nchilaka from comment #3)
> > I still see this as an issue, for eg take a case when a 100 gb file was 99gb
> > healed, then in this case we try to scan and heal from beginning.
> 
> For the record, this is something which can be fixed by introducing granular
> data-self-heal.

Or supporting general purpose sharding, which seems to be needed for DHT as well.

--- Additional comment from Ravishankar N on 2017-07-18 11:10:23 UTC ---

(In reply to Pranith Kumar K from comment #6)
> Or supporting general purpose sharding, which seems to be needed for DHT as
> well.

Yes that would work too. I guess whether we should *not* be doing granular data self-heal just because we can piggy back on files being striped is a discussion we need to have upstream.  I can imagine users who wouldn't want to stripe their files just because faster heal time is a by-product of the striping, if granular data-heal can guarantee faster heal times without it.