Bug 1059189

Summary:	nfs: remote operation failed: Directory not empty
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Saurabh <saujain>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED DUPLICATE	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.1	CC:	mketchion, mzywusko, nlevinki, sabansal, saujain, skoduri, spalai, vagarwal, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-30 09:48:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Saurabh 2014-01-29 11:56:30 UTC

Description of problem:
Well over a nfs mount of the glusterfs volume, I had earlier created lot of files and directories. 

I was trying to remove all of data from nfs mount point, but the problem is that two directories were still not deleted.

Meanwhile from fuse mount point I also started "ls -lR". 

Though listing the files/directories should not be a problem for deleting, hence
putting this issue as a bug.

Version-Release number of selected component (if applicable):
glusterfs-3.4.0.58rhs-1.el6rhs.x86_64

How reproducible:
happened this time 

Steps to Reproduce:
1. create a volume, start it
2. mount volume over nfs, say mount_point is /mnt/nfs-test
3. create lots of files and directories.
4. mount volume over fuse, say mount_point is /mnt/glusterfs-test
5. from /mnt/nfs-test, start executing "rm -rf *"
7. from /mnt/glusterfs-test, start executing "ls -lR"

Actual results:
two directories are still not deleted, although content inside is deleted,
[root@rhsauto005 nfs-regression.1390891943]# ls
quotadir  run5421

[root@rhsauto005 nfs-regression.1390891943]# ls run5421/
[root@rhsauto005 nfs-regression.1390891943]# ls quotadir/


glimpse of nfs.log,
[2014-01-28 13:01:41.665500] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-dist-rep-client-10: remote operation failed: Directory not empty
[2014-01-28 13:01:41.665608] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-dist-rep-client-11: remote operation failed: Directory not empty
[2014-01-28 13:01:41.667425] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-dist-rep-client-0: remote operation failed: File exists. Path: /run5421
[2014-01-28 13:01:41.667644] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-dist-rep-client-1: remote operation failed: File exists. Path: /run5421
[2014-01-28 13:01:41.667811] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-dist-rep-client-10: remote operation failed: File exists. Path: /run5421
[2014-01-28 13:01:41.667866] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-dist-rep-client-11: remote operation failed: File exists. Path: /run5421
[2014-01-28 13:01:41.672168] W [nfs3.c:3514:nfs3svc_rmdir_cbk] 0-nfs: ead1560a: /run5421 => -1 (Directory not empty)

Expected results:
Deletion should finish after deleting everything as we are using "rm -rf *" on the mount point

Additional info:
further trial of deletion with rm -rf still fails to delete dir "quotadir", as can be seen here,
[root@rhsauto005 nfs-regression.1390891943]# rm -rf *
rm: cannot remove `quotadir': Directory not empty

Comment 3 santosh pradhan 2014-01-30 05:51:50 UTC

Does it happen without "ls -lR" from FUSE mount?

If it just happens when rm -rf from NFS mount and ls -lR from FUSE mount, then its not a supported use case for now. So cant be a defect.

Comment 4 Saurabh 2014-01-30 10:42:40 UTC

(In reply to santosh pradhan from comment #3)
> Does it happen without "ls -lR" from FUSE mount?
> 
> If it just happens when rm -rf from NFS mount and ls -lR from FUSE mount,
> then its not a supported use case for now. So cant be a defect.

I have already mentioned as Additional Information that further trial of deletion of directory is not successful even if there is no other operation getting executed from any other mount-point.

Comment 5 Sakshi 2015-07-16 09:36:59 UTC

This looks like the race between rmdir and lookup selfheal. rmdir on "quotadir" is resulting in 'Directory not empty' mostly because, it still has child files/dir which are present on backend. These child dir/files are not visible from mount since they are deleted from hashed subvol. 

Can this be confirmed?

Comment 6 Vivek Agarwal 2015-07-28 13:37:07 UTC

Soumya, Can you answer the question in comment 5?

Comment 7 Soumya Koduri 2015-07-28 13:47:23 UTC

This was reported on glusterfs-3.4.0* version. I am not sure if this bug is still valid as gluster operations (which include removal of a directory) are tested on gluster-nfs mounts using latest builds. Saurabh, can you please confirm if this issue is reproducible on the latest RHGS builds?

Comment 8 Susant Kumar Palai 2015-11-30 09:48:35 UTC


*** This bug has been marked as a duplicate of bug 1115367 ***

Comment 9 Red Hat Bugzilla 2023-09-14 02:02:53 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days