Bug 785522 - Janitor should remove gfid handles
Summary: Janitor should remove gfid handles
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: posix
Version: pre-release
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Pranith Kumar K
QA Contact: Shwetha Panduranga
URL:
Whiteboard:
Depends On:
Blocks: 817967
TreeView+ depends on / blocked
 
Reported: 2012-01-29 10:41 UTC by Pranith Kumar K
Modified: 2013-07-24 17:34 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-24 17:34:44 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Description Pranith Kumar K 2012-01-29 10:41:41 UTC 
Description of problem:
Janitor should remove gfid handles.
Comment 1 Anand Avati 2012-01-30 07:29:50 UTC 
CHANGE: http://review.gluster.com/2697 (storage/posix: janitor should unset gfid handles) merged in master by Anand Avati (avati)
Comment 2 Pranith Kumar K 2012-06-01 12:12:46 UTC 
Have a replicate volume, start and mount it
On the mount create a directory with 10 entries.
Bring a brick down and delete that directory.
Bring the brick (down-brick) back up and do lookup on the deleted dir. check if the down-brick directory has .landfill and inside there will be a directory with the files that have been created. After 10 minutes that directory will be deleted by the janitor thread. Now see if gfid-backend has any files.
Comment 3 Shwetha Panduranga 2012-06-04 07:11:02 UTC 
Verified the fix on 3.3.0qa45. Bug is fixed. 

Steps executed:-
---------------
1. gluster volume create vol1 replica 3 10.16.159.184:/export_b1/dir1/ 10.16.159.188:/export_b1/dir1/ 10.16.159.196:/export_b1/dir1/
Creation of volume vol1 has been successful. Please start the volume to access data.

2. [06/04/12 - 02:34:15 root@AFR-Server1 ~]# gluster v info
 
Volume Name: vol1
Type: Replicate
Volume ID: c17f8341-7fc2-437e-8ceb-a8d887d233c1
Status: Created
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.16.159.184:/export_b1/dir1
Brick2: 10.16.159.188:/export_b1/dir1
Brick3: 10.16.159.196:/export_b1/dir1


3. [06/04/12 - 02:34:30 root@AFR-Server1 ~]# gluster v start vol1
Starting volume vol1 has been successful

4. [06/04/12 - 02:34:38 root@AFR-Server1 ~]# gluster v status
Status of volume: vol1
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 10.16.159.184:/export_b1/dir1			24009	Y	21802
Brick 10.16.159.188:/export_b1/dir1			24009	Y	27956
Brick 10.16.159.196:/export_b1/dir1			24009	Y	11779
NFS Server on localhost					38467	Y	21808
Self-heal Daemon on localhost				N/A	Y	21813
NFS Server on 10.16.159.188				38467	Y	27962
Self-heal Daemon on 10.16.159.188			N/A	Y	27968
NFS Server on 10.16.159.196				38467	Y	11785
Self-heal Daemon on 10.16.159.196			N/A	Y	11790

##### Create a fuse mount . On fuse mount create dir and files #####
5. [06/04/12 - 02:34:47 root@ARF-Client1 ~]# mount -t glusterfs 10.16.159.184:/vol1 /mnt/gfsc1

6. [06/04/12 - 02:37:49 root@ARF-Client1 gfsc1]# mkdir dir; for i in {1..10}; do dd if=/dev/urandom of=dir/file.$i bs=1M count=1; done

##### Bring down brick2 and brick3 ######
7. [06/04/12 - 02:42:10 root@AFR-Server1 ~]# gluster v status
Status of volume: vol1
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 10.16.159.184:/export_b1/dir1			24009	Y	21802
Brick 10.16.159.188:/export_b1/dir1			24009	N	27956
Brick 10.16.159.196:/export_b1/dir1			24009	N	11779
NFS Server on localhost					38467	Y	21808
Self-heal Daemon on localhost				N/A	Y	21813
NFS Server on 10.16.159.188				38467	Y	27962
Self-heal Daemon on 10.16.159.188			N/A	Y	27968
NFS Server on 10.16.159.196				38467	Y	11785
Self-heal Daemon on 10.16.159.196			N/A	Y	11790

##### From fuse mount remove the directory
8. [06/04/12 - 02:43:12 root@ARF-Client1 gfsc1]# rm -rf dir/

##### Bring Back the bricks #####
9. [06/04/12 - 02:43:43 root@AFR-Server1 ~]# gluster v start vol1 force
Starting volume vol1 has been successful

[06/04/12 - 02:44:11 root@AFR-Server1 ~]# gluster v status
Status of volume: vol1
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 10.16.159.184:/export_b1/dir1			24009	Y	21802
Brick 10.16.159.188:/export_b1/dir1			24009	Y	28020
Brick 10.16.159.196:/export_b1/dir1			24009	Y	11838
NFS Server on localhost					38467	Y	21907
Self-heal Daemon on localhost				N/A	Y	21913
NFS Server on 10.16.159.188				38467	Y	28026
Self-heal Daemon on 10.16.159.188			N/A	Y	28032
NFS Server on 10.16.159.196				38467	Y	11844
Self-heal Daemon on 10.16.159.196			N/A	Y	11850



Brick 2 ".landfill" directory before deletion of directory by janitor thread:-
----------------------------------------------------------------------------

[06/04/12 - 02:43:46 root@AFR-Server2 ~]# ls -lah /export_b1/dir1/
total 8.0K
drwxr-xr-x.  4 root root   39 Jun  4 02:43 .
drwxr-xr-x.  3 root root   17 Jun  4 02:33 ..
drw-------. 17 root root 4.0K Jun  4 02:38 .glusterfs
drwxr-xr-x.  3 root root   23 Jun  4 02:44 .landfill

[06/04/12 - 02:44:22 root@AFR-Server2 ~]# ls -lah /export_b1/dir1/.landfill/
total 8.0K
drwxr-xr-x. 3 root root   23 Jun  4 02:44 .
drwxr-xr-x. 4 root root   39 Jun  4 02:43 ..
drwxr-xr-x. 2 root root 4.0K Jun  4 02:38 3248166315

[06/04/12 - 02:44:47 root@AFR-Server2 ~]# ls -lah /export_b1/dir1/.landfill/3248166315/
total 11M
drwxr-xr-x. 2 root root 4.0K Jun  4 02:38 .
drwxr-xr-x. 3 root root   23 Jun  4 02:44 ..
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.1
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.10
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.2
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.3
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.4
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.5
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.6
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.7
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.8
-rw-r--r--. 2 root root 1.0M Jun  4 02:38 file.9

Brick 2 ".landfill" directory after deletion of dir by janitor thread:-
---------------------------------------------------------------------
[06/04/12 - 02:53:59 root@AFR-Server2 ~]# ls -la /export_b1/dir1/
total 8
drwxr-xr-x.  4 root root   39 Jun  4 02:43 .
drwxr-xr-x.  3 root root   17 Jun  4 02:33 ..
drw-------. 17 root root 4096 Jun  4 02:38 .glusterfs
drwxr-xr-x.  3 root root   23 Jun  4 02:44 .landfill

[06/04/12 - 02:54:13 root@AFR-Server2 ~]# ls -la /export_b1/dir1/.landfill/
total 4
drwxr-xr-x. 2 root root  6 Jun  4 02:54 .
drwxr-xr-x. 4 root root 39 Jun  4 02:43 ..
Comment 4 Shwetha Panduranga 2012-06-04 07:13:12 UTC 
Contents of .glusterfs after removal of dir from .landfill by janitor thread:-
------------------------------------------------------------------------------

[06/04/12 - 02:56:00 root@AFR-Server3 ~]# ls -lRa /export_b1/dir1/.glusterfs/
/export_b1/dir1/.glusterfs/:
total 8
drw-------. 17 root root 4096 Jun  4 02:38 .
drwxr-xr-x.  4 root root   39 Jun  4 02:43 ..
drwx------.  3 root root   15 Jun  4 02:34 00
drwx------.  3 root root   15 Jun  4 02:38 25
drwx------.  3 root root   15 Jun  4 02:38 52
drwx------.  3 root root   15 Jun  4 02:38 56
drwx------.  3 root root   15 Jun  4 02:38 63
drwx------.  3 root root   15 Jun  4 02:38 6e
drwx------.  3 root root   15 Jun  4 02:38 70
drwx------.  3 root root   15 Jun  4 02:37 80
drwx------.  3 root root   15 Jun  4 02:38 a7
drwx------.  3 root root   15 Jun  4 02:38 b8
drwx------.  3 root root   15 Jun  4 02:38 b9
drwx------.  3 root root   15 Jun  4 02:38 bc
drwx------.  3 root root   15 Jun  4 02:38 ec
drwx------.  3 root root   15 Jun  4 02:36 fc
drw-------.  3 root root   20 Jun  4 02:37 indices

/export_b1/dir1/.glusterfs/00:
total 4
drwx------.  3 root root   15 Jun  4 02:34 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root   49 Jun  4 02:34 00

/export_b1/dir1/.glusterfs/00/00:
total 0
drwx------. 2 root root 49 Jun  4 02:34 .
drwx------. 3 root root 15 Jun  4 02:34 ..
lrwxrwxrwx. 1 root root  8 Jun  4 02:34 00000000-0000-0000-0000-000000000001 -> ../../..

/export_b1/dir1/.glusterfs/25:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 ef

/export_b1/dir1/.glusterfs/25/ef:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/52:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 20

/export_b1/dir1/.glusterfs/52/20:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/56:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 3a

/export_b1/dir1/.glusterfs/56/3a:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/63:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 87

/export_b1/dir1/.glusterfs/63/87:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/6e:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 31

/export_b1/dir1/.glusterfs/6e/31:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/70:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 25

/export_b1/dir1/.glusterfs/70/25:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/80:
total 4
drwx------.  3 root root   15 Jun  4 02:37 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:37 11

/export_b1/dir1/.glusterfs/80/11:
total 0
drwx------. 2 root root  6 Jun  4 02:37 .
drwx------. 3 root root 15 Jun  4 02:37 ..

/export_b1/dir1/.glusterfs/a7:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 ad

/export_b1/dir1/.glusterfs/a7/ad:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/b8:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 42

/export_b1/dir1/.glusterfs/b8/42:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/b9:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:44 20

/export_b1/dir1/.glusterfs/b9/20:
total 0
drwx------. 2 root root  6 Jun  4 02:44 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/bc:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 69

/export_b1/dir1/.glusterfs/bc/69:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/ec:
total 4
drwx------.  3 root root   15 Jun  4 02:38 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:54 ab

/export_b1/dir1/.glusterfs/ec/ab:
total 0
drwx------. 2 root root  6 Jun  4 02:54 .
drwx------. 3 root root 15 Jun  4 02:38 ..

/export_b1/dir1/.glusterfs/fc:
total 4
drwx------.  3 root root   15 Jun  4 02:36 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drwx------.  2 root root    6 Jun  4 02:37 91

/export_b1/dir1/.glusterfs/fc/91:
total 0
drwx------. 2 root root  6 Jun  4 02:37 .
drwx------. 3 root root 15 Jun  4 02:36 ..

/export_b1/dir1/.glusterfs/indices:
total 4
drw-------.  3 root root   20 Jun  4 02:37 .
drw-------. 17 root root 4096 Jun  4 02:38 ..
drw-------.  2 root root    6 Jun  4 02:44 xattrop

/export_b1/dir1/.glusterfs/indices/xattrop:
total 0
drw-------. 2 root root  6 Jun  4 02:44 .
drw-------. 3 root root 20 Jun  4 02:37 ..
Note You need to log in before you can comment on or make changes to this bug.