Bug 988350

Summary: "volume heal info healed" command outputs healed files and directories information under sync nodes in case of disk replacement and heal triggered with "heal full" command
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: spandura
Component: replicateAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED WONTFIX QA Contact: spandura
Severity: medium Docs Contact:
Priority: medium    
Version: 2.1CC: nsathyan, rhs-bugs, storage-qa-internal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-11 05:25:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description spandura 2013-07-25 11:36:13 UTC
Description of problem:
=========================
When "gluster volume heal <volume_name> full" self-heal is triggered and subsequent "gluster volume heal <volume_name> info healed" command is executed to check what files are self-healed, the command outputs the file and directories which got self-healed under sync node. 

Expected results:
==================
The command should output files/directories which got self-healed under source node. 

Version-Release number of selected component (if applicable):
============================================================
root@rhs-client11 [Jul-25-2013-16:45:16] >rpm -qa | grep glusterfs-server
glusterfs-server-3.4.0.12rhs.beta6-1.el6rhs.x86_64

root@rhs-client11 [Jul-25-2013-16:45:22] >gluster --version
glusterfs 3.4.0.12rhs.beta6 built on Jul 23 2013 16:20:03

How reproducible:
====================
Often

Steps to Reproduce:
======================
1. Create a 6 x 2 distribute-replicate volume(4 storage nodes and 3 bricks on each storage node) . Start the volume

2. Create fuse mount. Create files and directories from fuse mount. 

3. on storage_node2 and storage_node4 killall gluster process. (killall glusterfs glusterfsd glusterd )

4. get the "trusted.glusterfs.volume-id" extended attribute for the brick directories on storage_node2 and storage_node4. (getfattr -d -e hex -m . <absolute_path_to_brick>)

5. delete the brick directories on storage_node2 and storage_node4. 

6. create the brick directories on storage_node2 and storage_node4.

7. set the "trusted.glusterfs.volume-id" extended attribute on all the bricks to the value captured in step 4. (setfattr -n  trusted.glusterfs.volume-id -v <value_of_extended_attribute> <absolute_path_to_brick>)

8. restart glusterd on storage_node2 and storage_node4. ( service glusterd start) 

9. trigger full self-heal : "gluster volume heal <volume_name> full" 

10. execute: "gluster volume heal <volume_name> info healed" 

Actual results:
================
In the following output "rhs-client11" and "rhs-client13" were the source nodes and "rhs-client12" and "rhs-client14" were the sync nodes. 

root@rhs-client11 [Jul-25-2013-16:38:06] >gluster v heal `gluster v list` info healed
Gathering heal info on volume dis_rep_vol1 has been successful

Brick rhs-client11:/rhs/brick1/b0
Number of entries: 0

Brick rhs-client12:/rhs/brick1/b1
Number of entries: 30
at                    path on brick
-----------------------------------
2013-07-25 11:07:53 /test_hardlink_self_heal
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.9
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.9
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/link_file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/file.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/file.9
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/link_file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1/new_file.7
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1/new_file.8
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.7
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.8
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.5

Brick rhs-client11:/rhs/brick1/b2
Number of entries: 0

Brick rhs-client12:/rhs/brick1/b3
Number of entries: 37
at                    path on brick
-----------------------------------
2013-07-25 11:07:53 /test_hardlink_self_heal
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.5
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.10
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.6
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.3
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.2/file.4
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.2/file.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.2/file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.2/link_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3/file.2
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3/file.3
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3/file.4
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3/file.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3/file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.3/link_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.4/file.7
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.4/file.8
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.4/link_file.2
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.7
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.8
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/link_file.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.3

Brick rhs-client11:/rhs/brick1/b4
Number of entries: 0

Brick rhs-client12:/rhs/brick1/b5
Number of entries: 25
at                    path on brick
-----------------------------------
2013-07-25 11:07:53 /test_hardlink_self_heal
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.6
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.6
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/file.6
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/link_file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/link_file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5/link_file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5/link_file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/new_dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/new_dir.1/new_file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/new_dir.1/new_file.5
2013-07-25 11:07:53 /test_hardlink_self_heal/new_dir.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.5
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3/new_file.9
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.9
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5

Brick rhs-client13:/rhs/brick1/b6
Number of entries: 0

Brick rhs-client14:/rhs/brick1/b7
Number of entries: 31
at                    path on brick
-----------------------------------
2013-07-25 11:07:53 /test_hardlink_self_heal
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/link_file.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/link_file.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/link_file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/link_file.10
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/link_file.8
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/link_file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1/new_file.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1/new_file.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.7
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.9

Brick rhs-client13:/rhs/brick1/b8
Number of entries: 0

Brick rhs-client14:/rhs/brick1/b9
Number of entries: 29
at                    path on brick
-----------------------------------
2013-07-25 11:07:53 /test_hardlink_self_heal
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/link_file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/link_file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/link_file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/link_file.7
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.9
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/link_file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5/file.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5/file.9
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.5/link_file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/new_dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/new_dir.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3/new_file.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3/new_file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4/new_file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.7
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5/new_file.8

Brick rhs-client13:/rhs/brick1/b10
Number of entries: 0

Brick rhs-client14:/rhs/brick1/b11
Number of entries: 35
at                    path on brick
-----------------------------------
2013-07-25 11:07:53 /test_hardlink_self_heal
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.1/link_file.10
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/link_file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.2/link_file.10
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/link_file.8
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.3/link_file.10
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.2
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.3
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.4
2013-07-25 11:07:53 /test_hardlink_self_heal/dir.4/file.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.4/file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.4/link_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.2
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.3
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.4
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.5
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/file.10
2013-07-25 11:07:54 /test_hardlink_self_heal/dir.5/link_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1/new_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.1/new_file.9
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.6
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.2/new_file.9
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3/new_file.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.3/new_file.5
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.4
2013-07-25 11:07:54 /test_hardlink_self_heal/new_dir.5
root@rhs-client11 [Jul-25-2013-16:38:08] >

Additional info:
===============

root@rhs-client11 [Jul-25-2013-17:01:30] >gluster v info
 
Volume Name: dis_rep_vol1
Type: Distributed-Replicate
Volume ID: 9db642e0-8c6f-407b-bc1e-62c8cc5b7f9c
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: rhs-client11:/rhs/brick1/b0
Brick2: rhs-client12:/rhs/brick1/b1
Brick3: rhs-client11:/rhs/brick1/b2
Brick4: rhs-client12:/rhs/brick1/b3
Brick5: rhs-client11:/rhs/brick1/b4
Brick6: rhs-client12:/rhs/brick1/b5
Brick7: rhs-client13:/rhs/brick1/b6
Brick8: rhs-client14:/rhs/brick1/b7
Brick9: rhs-client13:/rhs/brick1/b8
Brick10: rhs-client14:/rhs/brick1/b9
Brick11: rhs-client13:/rhs/brick1/b10
Brick12: rhs-client14:/rhs/brick1/b11

Comment 2 Scott Haines 2013-09-27 17:08:02 UTC
Targeting for 3.0.0 (Denali) release.

Comment 3 Nagaprasad Sathyanarayana 2014-05-06 11:43:39 UTC
Dev ack to 3.0 RHS BZs

Comment 5 spandura 2014-06-11 05:25:19 UTC
The command "gluster volume heal <volume_name> info healed" is not supported anymore from the gluster build : 

"[root@rhs-client11 ~]# gluster --version
glusterfs 3.6.0.15 built on Jun  9 2014 11:03:54"

Refer to bug : https://bugzilla.redhat.com/show_bug.cgi?id=1104486

Hence this bug is not valid anymore. Moving the bug to CLOSED state.