+++ This bug was initially created as a clone of Bug #1294675 +++
Description of problem:
From the command line of each host, and now constantly monitored by our Nagios/Centreon setup, we see that our 3 nodes replica-3 gluster storage volume is very frequently healing files, not to say constantly.
Version-Release number of selected component (if applicable):
Our setup : 3 Centos 7.2 nodes, with gluster 3.7.6 in replica-3, used as storage+compute for an oVirt 3.5.6 DC.
Install an oVirt setup on 3 nodes with glusterFS as direct gluster storage.
We have only 3 VMs running on it, so approx not more than 8 files (yes : only 8 files - the VM qemu files).
Steps to Reproduce:
1. Just run it and watch : all is nice
2. Run "gluster volume heal some_vol info" on random nodes
3. Read that more than zero files are getting healed
More than zero files are getting healed
I expected the "Number of entries" of every node to appear in the graph as a flat zero line, most of the times, except for the rare cases of node reboot, after which healing is launched and takes some minutes (sometimes hours) but is doing good.
At first, I found out that I forgot to bump up the cluster.op-version, but this has been done, everything rebooted and back to up.
But this DC is very lightly used, and I'm sure the gluster clients (that are the gluster nodes themselves) should read and write in a synchronous and proper way, not leading to any healing need.
Please see :
--- Additional comment from Pranith Kumar K on 2016-01-11 04:45:59 EST ---
hi Nicolas Ecarnot,
Thanks for raising the bug. "gluster volume heal <volname> info" is designed to be run one per the volume. If we run multiple processes it may lead to "Possibly undergoing heal" messages as the two try to take same locks and they will fail.
--- Additional comment from Nicolas Ecarnot on 2016-01-11 04:48:11 EST ---
(In reply to Pranith Kumar K from comment #1)
> hi Nicolas Ecarnot,
> Thanks for raising the bug. "gluster volume heal <volname> info" is
> designed to be run one per the volume. If we run multiple processes it may
> lead to "Possibly undergoing heal" messages as the two try to take same
> locks and they will fail.
Thank you Pranith for your answer.
Do you advice us to setup our Nagios/Centreon to run only *ONE* check per volume?
If so, please don't close this bug, let us change the setup, wait one week and I'll report the result here.
--- Additional comment from Pranith Kumar K on 2016-01-18 05:56:32 EST ---
hi Nicolas Ecarnot,
Sorry for the delay. Sure doing that will definitely help us. There could still be one corner case of self-heal-daemon and heal info conflicting for same locks. But I would like to hear more from you.
--- Additional comment from Nicolas Ecarnot on 2016-01-18 08:35:28 EST ---
(In reply to Pranith Kumar K from comment #3)
> hi Nicolas Ecarnot,
> Sorry for the delay. Sure doing that will definitely help us. There
> could still be one corner case of self-heal-daemon and heal info conflicting
> for same locks. But I would like to hear more from you.
On january 12, 2016, we modified our Nagios/Centreon to offset the checks of our 3 nodes'healing status.
2 weeks later, the graphs are showing a great decrease of healing cases, though not null.
This sounds encouraging.
Being recently noticed about sharding, this is the next feature to try and see whether it could improve the healing cases.
I let you decide if this is enough to close this bug - my opinion is that I'm still surprised that the healing cases is *not* constantly zero, but you choose.
REVIEW: http://review.gluster.org/13873 (cluster/afr: Fix spurious entries in heal info) posted (#1) for review on master by Pranith Kumar Karampuri (firstname.lastname@example.org)
REVIEW: http://review.gluster.org/13873 (cluster/afr: Fix spurious entries in heal info) posted (#2) for review on master by Pranith Kumar Karampuri (email@example.com)
REVIEW: http://review.gluster.org/13873 (cluster/afr: Fix spurious entries in heal info) posted (#3) for review on master by Pranith Kumar Karampuri (firstname.lastname@example.org)
COMMIT: http://review.gluster.org/13873 committed in master by Pranith Kumar Karampuri (email@example.com)
Author: Pranith Kumar K <firstname.lastname@example.org>
Date: Thu Mar 31 14:40:09 2016 +0530
cluster/afr: Fix spurious entries in heal info
Locking schemes in afr-v1 were locking the directory/file completely during
self-heal. Newer schemes of locking don't require Full directory, file locking.
But afr-v2 still has compatibility code to work-well with older clients, where
in entry-self-heal it takes a lock on a special 256 character name which can't
be created on the fs. Similarly for data self-heal there used to be a lock on
(LLONG_MAX-2, 1). Old locking scheme requires heal info to take sh-domain locks
before examining heal-state. If it doesn't take sh-domain locks, then there is
a possibility of heal-info hanging till self-heal completes because of
compatibility locks. But the problem with heal-info taking sh-domain locks is
that if two heal-info or shd, heal-info try to inspect heal state in parallel
using trylocks on sh-domain, there is a possibility that both of them assuming
a heal is in progress. This was leading to spurious entries being shown in
As long as there is afr-v1 way of locking, we can't fix this problem with
simple solutions. If we know that the cluster is running newer versions of
locking schemes, in those cases we can give accurate information in heal-info.
So introduce a new option called 'locking-scheme' which if it is 'granular'
will give correct information in heal-info. Not only that, Extra network hops
for taking compatibility locks, sh-domain locks in heal info will not be
necessary anymore. Thus it improves performance.
Signed-off-by: Pranith Kumar K <email@example.com>
Smoke: Gluster Build System <firstname.lastname@example.org>
NetBSD-regression: NetBSD Build System <email@example.com>
CentOS-regression: Gluster Build System <firstname.lastname@example.org>
Reviewed-by: Ashish Pandey <email@example.com>
Reviewed-by: Anuradha Talur <firstname.lastname@example.org>
Reviewed-by: Krutika Dhananjay <email@example.com>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.
glusterfs-3.8.0 has been announced on the Gluster mailinglists , packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist  and the update infrastructure for your distribution.