Description of problem:

Tried to remove a empty directory from NFS mount but it failed with "rm: cannot remove ‘bucket_version/’: Directory not empty" error.

Created some buckets with data using s3 some days before, and removed all data and there was only empty directory. 


Version-Release number of selected component (if applicable):

ceph version: 12.2.0-1.el7cp
nfs-ganesha version: 
nfs-ganesha-2.5.2-1.el7cp.x86_64
nfs-ganesha-rgw-2.5.2-1.el7cp.x86_64


How reproducible:
1/1


[root@magna071 mnt1]# cd bucket_version/
[root@magna071 bucket_version]# ls
[root@magna071 bucket_version]# ll -h
total 0
[root@magna071 bucket_version]# cd .. 
[root@magna071 mnt1]# ls
bucket1  bucket_version

[root@magna071 mnt1]# date
Wed Sep  6 07:01:12 UTC 2017
[root@magna071 mnt1]# rm -rf *
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# ls
bucket_version
[root@magna071 mnt1]# rm -rf *
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# 


After updating packages, umounting, rebooting and remounting also faced the same error.


[root@magna071 ubuntu]# umount  -l mnt1
[root@magna071 ubuntu]# 
[root@magna071 ubuntu]# reboot
Connection to magna071 closed by remote host.
Connection to magna071 closed.
vidushi@magna002:~$ ssh magna071
Warning: Permanently added 'magna071,10.8.128.71' (ECDSA) to the list of known hosts.
Last login: Wed Sep  6 16:38:05 2017 from 10.8.128.2
[ubuntu@magna071 ~]$ sudo -s
[root@magna071 ubuntu]# mount -t nfs -o vers=3 magna100:/ /home/ubuntu/mnt1
[root@magna071 ubuntu]# cd mnt1
[root@magna071 mnt1]# ls
bucket_1  bucket_version  bucket_version_1  bucket_version_new
[root@magna071 mnt1]#  
[root@magna071 mnt1]# rm -rf  bucket_version
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# rm -rf  bucket_version_new
rm: cannot remove ‘bucket_version_new’: Directory not empty
[root@magna071 mnt1]# 


Steps to Reproduce:
1. Configure Cluster with RGW and NFS
2. Create some directories with data using S3 and directory on NFS mount.
3. Perform RW IO to folders and buckets using different interface(S3 or kernel)
4. Delete all the files in the directory
5. Update the packages, unmount NFS, reboot the machine and remount it.
6. Try to remove the director

Actual results:
Deleting directories fails

Expected results:
Directory should be get deleted.

Additional info:

It is failing for some directories and passing for some.
Similar to bug https://bugzilla.redhat.com/show_bug.cgi?id=1422822