Bug 1473668

Summary:	Disperse: entries in heal info not getting healed at all
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nag Pavan Chilakam <nchilaka>
Component:	disperse	Assignee:	Sunil Kumar Acharya <sheggodu>
Status:	CLOSED CANTFIX	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	urgent	Docs Contact:
Priority:	medium
Version:	rhgs-3.3	CC:	amukherj, aspandey, pkarampu, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-06 14:14:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nag Pavan Chilakam 2017-07-21 12:14:04 UTC

Description of problem:
======================
Some set of files showing as entries in heal info are not at all getting healed.
I tried even manual heal triggering, but that too doesn't work.
The healing can only happens from client action that too if we try to append to the file (reads also didn't heal it)

I checked backend for one of the entries and below are the xattr settings
(all bricks in the 1x(4+2) shows the same details(with dirty pending marks)

the file is showing heal info as the file is available in indices/xattrops

[root@dhcp35-45 nfs]# ll /rhs/brick1/ecv/.glusterfs/indices/xattrop/|grep eff9
----------. 6 root root 0 Jul 21 12:56 dc1ae434-1fb7-42b8-8b21-514f3845eff9



b1:
getfattr -d -m . -e hex /rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000010000000000000000
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x00000000000000000000000000000000
trusted.gfid=0xdc1ae4341fb742b88b21514f3845eff9

b2:
[root@dhcp35-130 nfs]# getfattr -d -m . -e hex /rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000010000000000000000
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x00000000000000000000000000000000
trusted.gfid=0xdc1ae4341fb742b88b21514f3845eff9

b3:
[root@dhcp35-122 nfs]# getfattr -d -m . -e hex /rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000010000000000000000
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x00000000000000000000000000000000
trusted.gfid=0xdc1ae4341fb742b88b21514f3845eff9

b4:
[root@dhcp35-23 nfs]# getfattr -d -m . -e hex /rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000010000000000000000
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x00000000000000000000000000000000
trusted.gfid=0xdc1ae4341fb742b88b21514f3845eff9


b5:

[root@dhcp35-112 nfs]# getfattr -d -m . -e hex /rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000010000000000000000
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x00000000000000000000000000000000
trusted.gfid=0xdc1ae4341fb742b88b21514f3845eff9

b6:
[root@dhcp35-138 nfs]# getfattr -d -m . -e hex /rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/ecv//nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000010000000000000000
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x00000000000000000000000000000000
trusted.gfid=0xdc1ae4341fb742b88b21514f3845eff9


[root@dhcp35-45 ~]# gluster v heal ecv 
Launching heal operation to perform index self heal on volume ecv has been successful 
Use heal info commands to check status
[root@dhcp35-45 ~]# gluster v heal ecv info
Brick 10.70.35.45:/rhs/brick1/ecv
<gfid:a201cb23-ed16-47b3-8c4e-3566a96d9a80> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
<gfid:94d579be-9074-4bf0-a0b7-e3bd213662be> 
<gfid:b2edf1f1-a0df-4908-a769-54b6e704245e> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
Status: Connected
Number of entries: 5

Brick 10.70.35.130:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
<gfid:a201cb23-ed16-47b3-8c4e-3566a96d9a80> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
<gfid:94d579be-9074-4bf0-a0b7-e3bd213662be> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
<gfid:b2edf1f1-a0df-4908-a769-54b6e704245e> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.122:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.23:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.112:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.138:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

[root@dhcp35-45 ~]# gluster v heal ecv 
Launching heal operation to perform index self heal on volume ecv has been successful 
Use heal info commands to check status
[root@dhcp35-45 ~]# gluster v heal ecv info
Brick 10.70.35.45:/rhs/brick1/ecv
<gfid:a201cb23-ed16-47b3-8c4e-3566a96d9a80> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
<gfid:94d579be-9074-4bf0-a0b7-e3bd213662be> 
<gfid:b2edf1f1-a0df-4908-a769-54b6e704245e> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
Status: Connected
Number of entries: 5

Brick 10.70.35.130:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
<gfid:a201cb23-ed16-47b3-8c4e-3566a96d9a80> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
<gfid:94d579be-9074-4bf0-a0b7-e3bd213662be> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
<gfid:b2edf1f1-a0df-4908-a769-54b6e704245e> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.122:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.23:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.112:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.138:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

[root@dhcp35-45 ~]# cd /var/log/glusterfs/
[root@dhcp35-45 glusterfs]# gluster v heal ecv 
Launching heal operation to perform index self heal on volume ecv has been successful 
Use heal info commands to check status
[root@dhcp35-45 glusterfs]# gluster v heal ecv info
Brick 10.70.35.45:/rhs/brick1/ecv
<gfid:a201cb23-ed16-47b3-8c4e-3566a96d9a80> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
<gfid:94d579be-9074-4bf0-a0b7-e3bd213662be> 
<gfid:b2edf1f1-a0df-4908-a769-54b6e704245e> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
Status: Connected
Number of entries: 5

Brick 10.70.35.130:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
<gfid:a201cb23-ed16-47b3-8c4e-3566a96d9a80> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
<gfid:94d579be-9074-4bf0-a0b7-e3bd213662be> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
<gfid:b2edf1f1-a0df-4908-a769-54b6e704245e> 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.122:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.23:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.112:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

Brick 10.70.35.138:/rhs/brick1/ecv
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nspire.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/mrvl.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/tauros2.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/mrvl/feroceon.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/qcom,idle-state.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/timer.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/nxp/lpc32xx.txt 
/nfs/linux-4.12.3/Documentation/devicetree/bindings/arm/msm/ssbi.txt 
Status: Connected
Number of entries: 9

[root@dhcp35-45 glusterfs]# gluster v info 
 
Volume Name: ecv
Type: Disperse
Volume ID: 0c2bd796-527f-4dd7-a1c1-a5175470140a
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.35.45:/rhs/brick1/ecv
Brick2: 10.70.35.130:/rhs/brick1/ecv
Brick3: 10.70.35.122:/rhs/brick1/ecv
Brick4: 10.70.35.23:/rhs/brick1/ecv
Brick5: 10.70.35.112:/rhs/brick1/ecv
Brick6: 10.70.35.138:/rhs/brick1/ecv
Options Reconfigured:
nfs.disable: off
transport.address-family: inet
[root@dhcp35-45 glusterfs]# gluster v status
Status of volume: ecv
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick1/ecv           49152     0          Y       29478
Brick 10.70.35.130:/rhs/brick1/ecv          49153     0          Y       30084
Brick 10.70.35.122:/rhs/brick1/ecv          49152     0          Y       25556
Brick 10.70.35.23:/rhs/brick1/ecv           49152     0          Y       13250
Brick 10.70.35.112:/rhs/brick1/ecv          49152     0          Y       22746
Brick 10.70.35.138:/rhs/brick1/ecv          49152     0          Y       3535 
NFS Server on localhost                     2049      0          Y       29458
Self-heal Daemon on localhost               N/A       N/A        Y       29468
NFS Server on 10.70.35.112                  2049      0          Y       22782
Self-heal Daemon on 10.70.35.112            N/A       N/A        Y       22766
NFS Server on 10.70.35.122                  2049      0          Y       25643
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       25576
NFS Server on 10.70.35.23                   2049      0          Y       13286
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       13270
NFS Server on 10.70.35.138                  2049      0          Y       3572 
Self-heal Daemon on 10.70.35.138            N/A       N/A        Y       3555 
NFS Server on 10.70.35.130                  2049      0          Y       30065
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       30074
 
Task Status of Volume ecv
------------------------------------------------------------------------------
There are no active volume tasks



Version-Release number of selected component (if applicable):
======
3.8.4-34



Steps to Reproduce:
1.created a 1x(4+2) volume , the intent being to test in-service upgrade as part of validating bz#1465289
2.mounted volume on 2 different clients 1 gnfs and 1 fuse
3.created seperate working directories for each protocol and copied linux image for untar
4. started linux untar from both images( may be not parallely, but a few seconds or max 2 min delay between each untar)
5. kill glusterfsd,glusterfs and glusterd stop of n1(hosting b1) so as to upgrade
6. after 5 min or so, ran step#5 on n2
7. didn't go ahead with upgrade as new rpms were not available.
8. brought up glusterd on n1 and n2
8. healing triggered and all files other than about 10 files healed.

However the ~10 files are not at all healing, even after 5 hrs

Comment 2 Nag Pavan Chilakam 2017-07-21 14:26:21 UTC

sosreports @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1473668

Comment 3 Nag Pavan Chilakam 2017-07-25 10:16:05 UTC

proposing as blocker as it is a serious problem

Comment 11 Sunil Kumar Acharya 2017-12-06 14:14:12 UTC

Relevant documentation is done as part of RHGS-3.3.0. Bug 1481946 is "CLOSED CURRENTRELEASE".