Description of problem:
=======================
I see that even after a day heal is not completing for one particular entry.
Hit the issue when post upgrading first of the three nodes, from 6.0.14 to 6.0.15.



Version-Release number of selected component (if applicable):
============
6.0.14 to 6.0.15 upgrade

How reproducible:
===============
hit it once


Steps to Reproduce:
==================
1) 3 node cluster with bmux enabled
2) one 1x3 volume and one 20x(4+2) ecvol(see vol info details)
3) triggered below IOs, on total of 5 clients:
 a) linux untar from 4 clients on ec-volume
 b) top o/p of clients appending to files on afrvolume
 c) crefi on 2 clients on ec-volume:
for j in {1..100};do for i in {create,chmod,chown,chgrp,symlink,truncate,rename,hardlink}; do python crefi.py --multi -n 5 -b 20 -d 10 --max=1K --min=50 --random -T 2 -t text --fop=$i /mnt/cvlt-async/IOs/crefi/$HOSTNAME/ ; sleep 10 ; done;done
 d) lookups while true;find *|xargs stat from all clients on ec-vol

4) after about 2 days, triggered inservice upgrade from 6.0.14 to 6.0.15 on first node.
5) heal info is showing below entry as pending for heal even after 1 day and hence unable to upgrade the whole cluster completely

note: bricks were down as part of upgrade for hardly less than 1hr at max

Actual results:
================
heal info is showing below entry as pending for heal even after 1 day and hence unable to upgrade the whole cluster completely

[root@rhs-gp-srv1 ~]# gluster v heal cvlt-async info
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16
/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 
Status: Connected
Number of entries: 1

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16
/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 
Status: Connected
Number of entries: 1

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16
/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 
Status: Connected
Number of entries: 1

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16
/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 
Status: Connected
Number of entries: 1

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20
Status: Connected
Number of entries: 0

Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20
Status: Connected
Number of entries: 0

Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20
Status: Connected
Number of entries: 0

Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20
Status: Connected
Number of entries: 0


Expected results:
================
heal should complete

Additional info:
===============
volinfo
Volume Name: cvlt-async
Type: Distributed-Disperse
Volume ID: 7cba899d-5642-44b9-a05b-57d78247d5ea
Status: Started
Snapshot Count: 0
Number of Bricks: 20 x (4 + 2) = 120
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1
Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1
Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1
Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1
Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2
Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2
Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2
Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2
Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2
Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2
Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3
Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3
Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3
Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3
Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3
Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3
Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4
Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4
Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4
Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4
Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4
Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4
Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5
Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5
Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5
Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5
Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5
Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5
Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6
Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6
Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6
Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6
Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6
Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6
Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7
Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7
Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7
Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7
Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7
Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7
Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8
Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8
Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8
Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8
Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8
Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8
Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9
Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9
Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9
Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9
Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9
Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9
Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10
Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10
Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10
Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10
Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10
Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10
Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11
Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11
Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11
Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11
Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11
Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11
Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12
Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12
Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12
Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12
Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12
Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12
Brick73: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13
Brick74: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13
Brick75: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13
Brick76: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13
Brick77: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13
Brick78: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13
Brick79: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14
Brick80: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14
Brick81: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14
Brick82: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14
Brick83: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14
Brick84: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14
Brick85: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15
Brick86: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15
Brick87: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15
Brick88: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15
Brick89: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15
Brick90: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15
Brick91: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16
Brick92: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16
Brick93: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16
Brick94: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16
Brick95: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16
Brick96: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16
Brick97: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17
Brick98: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17
Brick99: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17
Brick100: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17
Brick101: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17
Brick102: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17
Brick103: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18
Brick104: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18
Brick105: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18
Brick106: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18
Brick107: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18
Brick108: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18
Brick109: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19
Brick110: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19
Brick111: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19
Brick112: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19
Brick113: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19
Brick114: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19
Brick115: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20
Brick116: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20
Brick117: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20
Brick118: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20
Brick119: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20
Brick120: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20
Options Reconfigured:
features.ctime: enable
cluster.disperse-self-heal-daemon: enable
disperse.shd-max-threads: 24
client.event-threads: 8
server.event-threads: 8
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
cluster.brick-multiplex: enable
 
Volume Name: repvol
Type: Replicate
Volume ID: 5566101e-5321-4cdf-b560-7804b7addd58
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick44/repvol
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick44/repvol
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick44/repvol
Options Reconfigured:
cluster.self-heal-daemon: enable
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
cluster.brick-multiplex: enable




getfattr details of pending heal:
================================

good bricks of n1:
------------------
[root@rhs-gp-srv1 glusterfs]# getfattr -d -m . -e hex /gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.dirty=0x00000000000000050000000000000000
trusted.ec.version=0x00000000000000a100000000000000a8
trusted.gfid=0x991669123a0d486ba59284fe9e6784d5
trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48

[root@rhs-gp-srv1 glusterfs]# getfattr -d -m . -e hex /gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.dirty=0x00000000000000050000000000000000
trusted.ec.version=0x00000000000000a100000000000000a8
trusted.gfid=0x991669123a0d486ba59284fe9e6784d5
trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48



good bricks of n2:
================
[root@rhs-gp-srv2 glusterfs]# getfattr -d -m . -e hex /gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.dirty=0x00000000000000050000000000000000
trusted.ec.version=0x00000000000000a100000000000000a8
trusted.gfid=0x991669123a0d486ba59284fe9e6784d5
trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48

[root@rhs-gp-srv2 glusterfs]# getfattr -d -m . -e hex /gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.dirty=0x00000000000000050000000000000000
trusted.ec.version=0x00000000000000a100000000000000a8
trusted.gfid=0x991669123a0d486ba59284fe9e6784d5
trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48



bricks of node n3(which was upgraded):
------------------------------
[root@rhs-gp-srv4 glusterfs]# getfattr -d -m . -e hex /gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.version=0x000000000000002500000000000000a8
trusted.gfid=0x991669123a0d486ba59284fe9e6784d5
trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48

[root@rhs-gp-srv4 glusterfs]# getfattr -d -m . -e hex /gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.version=0x000000000000002500000000000000a8
trusted.gfid=0x991669123a0d486ba59284fe9e6784d5
trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48