761959 – (GLUSTER-227) replicate selfheal does not remove directory with contents in it

Bug 761959 (GLUSTER-227) - replicate selfheal does not remove directory with contents in it

Summary: replicate selfheal does not remove directory with contents in it

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	GLUSTER-227
Product:	GlusterFS
Classification:	Community
Component:	replicate
Sub Component:
Version:	mainline
Hardware:	All
OS:	Linux
Priority:	low
Severity:	low
Target Milestone:	---
Assignee:	Vikas Gorur
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	GLUSTER-386
TreeView+	depends on / blocked

Reported:	2009-08-25 11:51 UTC by Krishna Srinivas
Modified:	2010-04-07 09:42 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression:	RTA
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Krishna Srinivas 2009-08-25 11:51:08 UTC

ticket 1148 1211

This is a known issue. We plan to fix this in 2.1.

Comment 1 Anand Avati 2009-10-29 14:17:22 UTC

PATCH: http://patches.gluster.com/patch/2054 in master (cluster/afr: Move deleted files to /.trash in entry self-heal.)

Comment 2 Anand Avati 2009-12-02 15:29:25 UTC

PATCH: http://patches.gluster.com/patch/2491 in master (storage/posix: Added janitor thread.)

Comment 3 Anand Avati 2009-12-03 07:00:45 UTC

PATCH: http://patches.gluster.com/patch/2507 in master (storage/posix: Change janitor sleep duration to 10 minutes.)

Comment 4 Anand Avati 2009-12-03 08:44:08 UTC

PATCH: http://patches.gluster.com/patch/2520 in master (core: Change replicate trash directory name to ".landfill".)

Comment 5 Harshavardhana 2009-12-06 01:04:39 UTC

Created 100 files with size each 1GB -- Total 100GB . 

Shutdown the first subvolume delete all the files with second subvolume still 
there. 

Now start the first subvolume. 

Run "ls" in the directory self-heal starts and deletes almost all the files 
but leaves 11 files

-rw-r--r-- 1 root root 1073741824 Dec  5 13:28 big-file24
-rw-r--r-- 1 root root 1073741824 Dec  5 13:30 big-file28
-rw-r--r-- 1 root root 1073741824 Dec  5 13:32 big-file34
-rw-r--r-- 1 root root 1073741824 Dec  5 13:35 big-file42
-rw-r--r-- 1 root root 1073741824 Dec  5 13:17 big-file6
-rw-r--r-- 1 root root 1073741824 Dec  5 13:43 big-file62
-rw-r--r-- 1 root root 1073741824 Dec  5 13:47 big-file72
-rw-r--r-- 1 root root 1073741824 Dec  5 13:51 big-file85
-rw-r--r-- 1 root root 1073741824 Dec  5 13:52 big-file87
-rw-r--r-- 1 root root 1073741824 Dec  5 13:54 big-file90
-rw-r--r-- 1 root root 1073741824 Dec  5 13:54 big-file91

Relevant log files from first subvolume
-----------------------------
[2009-12-05 20:54:34] E [posix.c:619:posix_setattr] posix2: setattr (lstat) on /exports/sdb1/mirror/harsha/afr_test/big-file1 failed: No such file or directory
[2009-12-05 20:54:39] E [posix.c:619:posix_setattr] posix2: setattr (lstat) on /exports/sdb1/mirror/harsha/afr_test/big-file14 failed: No such file or directory
[2009-12-05 20:54:41] E [posix.c:619:posix_setattr] posix2: setattr (lstat) on /exports/sdb1/mirror/harsha/afr_test/big-file2 failed: No such file or directory
[2009-12-05 20:54:42] E [posix.c:619:posix_setattr] posix2: setattr (lstat) on /exports/sdb1/mirror/harsha/afr_test/big-file4 failed: No such file or directory
[2009-12-05 20:54:42] E [posix.c:619:posix_setattr] posix2: setattr (lstat) on /exports/sdb1/mirror/harsha/afr_test/big-file5 failed: No such file or directory
[2009-12-05 20:54:43] E [posix.c:619:posix_setattr] posix2: setattr (lstat) on /exports/sdb1/mirror/harsha/afr_test/big-file9 failed: No such file or directory
[2009-12-05 20:54:44] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file13 failed: No such file or directory
[2009-12-05 20:54:45] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file10 failed: No such file or directory
[2009-12-05 20:54:45] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file11 failed: No such file or directory
[2009-12-05 20:54:47] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file12 failed: No such file or directory
[2009-12-05 20:54:47] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file16 failed: No such file or directory
[2009-12-05 20:54:48] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file15 failed: No such file or directory
[2009-12-05 20:54:48] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file3 failed: No such file or directory
[2009-12-05 20:54:49] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file6 failed: No such file or directory
[2009-12-05 20:54:49] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file7 failed: No such file or directory
[2009-12-05 20:54:49] E [posix.c:619:posix_setattr] posix1: setattr (lstat) on /exports/sdb2/mirror/harsha/afr_test/big-file8 failed: No such file or directory
------------------------

Comment 6 Harshavardhana 2009-12-06 01:09:35 UTC

Few more relevant logs i think would help. 

----------------------------
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file54 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file98 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file100 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file92 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file89 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file79 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file45 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file29 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file94 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file60 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file15 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file30 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file12 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file99 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file83 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file49 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file16 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file33 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file64 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file25 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file56 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file82 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file70 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file80 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file36 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file10 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file63 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file68 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file11 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file13 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file7 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file81 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file96 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file95 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file40 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file3 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix1: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file8 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file46 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file32 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file38 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file14 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file52 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file37 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file9 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file55 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file44 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file17 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file61 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file73 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file51 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file88 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file2 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file97 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file75 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file22 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file21 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file78 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file93 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file19 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file23 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file67 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file27 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file69 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file47 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file35 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file58 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file57 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file5 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file31 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file76 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file71 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file74 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file41 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file48 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file65 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file66 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file53 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file18 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file77 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file26 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file86 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file20 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file59 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file50 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file39 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file84 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file43 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file1 failed: No such file or directory
[2009-12-06 01:02:30] E [posix.c:477:posix_lookup] posix2: post-operation lstat on parent of .landfill/harsha-afr_test-dd_files-big-file4 failed: No such file or directory
---------------------------

Comment 7 Vikas Gorur 2009-12-07 10:30:27 UTC

This is a consequence of entry self-heal being done in the background. While the unlink is going on a lookup has triggered missing entry self-heal and recreated the files. A patch to disable background self-heal for entries has been sent for review.

Note You need to log in before you can comment on or make changes to this bug.