Bug 866908 - [RHEV-RHS] "gluster volume heal <vol_name> info" command outputs entries which are in ".glusterfs" and ".landfill" directory
[RHEV-RHS] "gluster volume heal <vol_name> info" command outputs entries whic...
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
Unspecified Unspecified
medium Severity unspecified
: ---
: ---
Assigned To: Pranith Kumar K
Depends On:
  Show dependency treegraph
Reported: 2012-10-16 05:52 EDT by spandura
Modified: 2012-12-13 01:08 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: Whenever a brick is restarted and self-heal has to remove a directory 'dir' it sets a flag indicating in rmdir fop that it is 'rm -rf'. The 'dir' is moved to '.glusterfs/landfill' by posix because of the flags in rmdir fop, Janitor thread removes it asynchronously. gfid-handles exists for 'dir' and files under 'dir' until the janitor thread deletes them from landfill. Janitor thread is run every 10 minutes. Self-heald depends on gfid-handle presence to determine that a file exists in the filesystem. Because of this behaviour, stale entries show up in the output of 'gluster volume heal <volname> info'. The issue is transient, after 10 minutes the stale entries won't show up in the output. Consequence: Workaround (if any): Wait for 10 minutes. The entries will be removed from the landfill directory. Volume heal <volname> will not show these stale entries. Result:
Story Points: ---
Clone Of:
Last Closed: 2012-12-13 01:08:20 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description spandura 2012-10-16 05:52:17 EDT
Description of problem:
"gluster volume heal <vol_name> info" command outputs entries which are in ".glusterfs" and ".landfill" directory

Version-Release number of selected component (if applicable):
[10/15/12 - 21:18:25 root@rhs-client6 ~]# rpm -qa | grep gluster

[10/15/12 - 21:18:58 root@rhs-client6 ~]# gluster --version
glusterfs 3.3.0rhsvirt1 built on Oct  8 2012 15:23:00

Steps to Reproduce:
1.Create a pure replicate volume (1x2) with 2 servers and 1 brick on each server. This is the storage for the VM's. start the volume.

2.Set-up the KVM to use the volume as VM store. 

3.Create 5 virtual machines (vm1 and vm2) 

4.power off both server1 and server2  

5.Poweron server2 immediately.

6.Start the VM's

7.execute "gluster volume heal <volume_name> info"   
Actual results:
[10/16/12 - 14:32:37 root@rhs-client6 ~]# gluster v heal replicate info
Heal operation on volume replicate has been successful

Brick rhs-client6.lab.eng.blr.redhat.com:/disk0
Number of entries: 4

Brick rhs-client7.lab.eng.blr.redhat.com:/disk0
Number of entries: 2

[10/16/12 - 14:34:19 root@rhs-client6 ~]# stat -c %i  /disk0/.glusterfs/86/2f/862f996a-29cf-44ed-a465-67c7953bea31

[10/16/12 - 14:34:31 root@rhs-client6 ~]# find /disk0 -inum 536871060 -print

[10/16/12 - 14:34:59 root@rhs-client6 ~]# stat -c %i /disk0/.glusterfs/9e/ad/9eadcb00-ff79-4884-b60d-2121133c3b45 

[10/16/12 - 14:35:17 root@rhs-client6 ~]# find /disk0 -inum 536871093 -print

[10/16/12 - 14:35:30 root@rhs-client6 ~]# stat -c %i /disk0/.glusterfs/09/11/0911332e-f679-4566-877f-778ce1863174 

[10/16/12 - 14:35:43 root@rhs-client6 ~]# find /disk0 -inum 1610612909 -print

Expected results:
we would expect heal info not to list ".glusterfs" and ".landfill" directory entries. 

Not sure of what is the exact expected result. 

Additional info:
root@rhs-client6 ~]# gluster volume info replicate
Volume Name: replicate
Type: Replicate
Volume ID: b2f1fd96-fcec-4110-81e6-963dba306d00
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: rhs-client6.lab.eng.blr.redhat.com:/disk0
Brick2: rhs-client7.lab.eng.blr.redhat.com:/disk0
Options Reconfigured:
storage.linux-aio: enable
cluster.eager-lock: enable
performance.read-ahead: disable
performance.stat-prefetch: disable
performance.io-cache: disable
performance.quick-read: disable
Comment 2 Pranith Kumar K 2012-11-01 03:21:30 EDT
Steps to recreate the issue:
1) Create a replicate volume with eager-lock enabled.
2) create a directory 'dir' and cd into it
3) start an infinite dd. 'dd of=a if=/dev/urandom' <--> Actually any command which can lead to index file in both the bricks for a noticeable time. When eager-lock is enabled, the chances of this happening increases.
4) kill one of the bricks
5) stop the dd using ctrl+c
6) rm -rf the directory 'dir'
7) do gluster volume start <volname> force
8) gluster volume heal <volname> info will show the stale file gfid entry on the brick which just came up.

[root@pranithk-laptop r2]# gluster volume heal r2 info
Gathering Heal info on volume r2 has been successful

Brick pranithk-laptop:/gfs/r2_0
Number of entries: 1

Brick pranithk-laptop:/gfs/r2_1
Number of entries: 0
[root@pranithk-laptop r2]# ls /gfs/r2_0/.glusterfs/landfill/
[root@pranithk-laptop r2]# ls /gfs/r2_0/.glusterfs/landfill/d785558b-9dfc-4750-9687-6704fe71cbef/
[root@pranithk-laptop r2]# ls /gfs/r2_0/.glusterfs/landfill/d785558b-9dfc-4750-9687-6704fe71cbef/a ^C
[root@pranithk-laptop r2]# getfattr -d -m . -e hex /gfs/r2_0/.glusterfs/landfill/d785558b-9dfc-4750-9687-6704fe71cbef/a 
getfattr: Removing leading '/' from absolute path names
# file: gfs/r2_0/.glusterfs/landfill/d785558b-9dfc-4750-9687-6704fe71cbef/a

Reason for the issue:
When the brick is restarted again, self-heal has to remove the directory 'dir' which is moved to landfill by posix because of the flags indicated by self-heal in rmdir fop. Janitor thread is run every 10 minutes, until then the stale entry shows up in the output of 'gluster volume heal <volname> info' Because the gfid-handle exists until the janitor thread deletes the stale entries in landfill.

Note You need to log in before you can comment on or make changes to this bug.