Bug 1489301

Summary: [RGW:NFS]: Directories not getting removed from NFS mount specifically when versioning is enabled
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vidushi Mishra <vimishra>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED NOTABUG QA Contact: Vidushi Mishra <vimishra>
Severity: high Docs Contact: Bara Ancincova <bancinco>
Priority: high    
Version: 3.0CC: anharris, bigoocare, cbodley, ceph-eng-bugs, hnallurv, kbader, mbenjamin, pasik, rperiyas, sweil, tchandra, uboppana
Target Milestone: rcKeywords: Regression
Target Release: 4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
.It is not possible to remove directories stored on S3 versioned buckets by using `rm` The mechanism that is used to check for non-empty directories prior to unlinking them works incorrectly in combination with the Ceph Object Gateway Simple Storage Service (S3) versioned buckets. As a consequence, directory trees on versioned buckets cannot be recursively removed with a command such as `rm -rf`. To work around this problem, remove any objects in versioned buckets by using the S3 interface.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-11 16:38:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1494421    

Description Vidushi Mishra 2017-09-07 07:31:05 UTC 
Description of problem:

Tried to remove a empty directory from NFS mount but it failed with "rm: cannot remove ‘bucket_version/’: Directory not empty" error.

Created some buckets with data using s3 some days before, and removed all data and there was only empty directory. 


Version-Release number of selected component (if applicable):

ceph version: 12.2.0-1.el7cp
nfs-ganesha version: 
nfs-ganesha-2.5.2-1.el7cp.x86_64
nfs-ganesha-rgw-2.5.2-1.el7cp.x86_64


How reproducible:
1/1


[root@magna071 mnt1]# cd bucket_version/
[root@magna071 bucket_version]# ls
[root@magna071 bucket_version]# ll -h
total 0
[root@magna071 bucket_version]# cd .. 
[root@magna071 mnt1]# ls
bucket1  bucket_version

[root@magna071 mnt1]# date
Wed Sep  6 07:01:12 UTC 2017
[root@magna071 mnt1]# rm -rf *
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# ls
bucket_version
[root@magna071 mnt1]# rm -rf *
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# 


After updating packages, umounting, rebooting and remounting also faced the same error.


[root@magna071 ubuntu]# umount  -l mnt1
[root@magna071 ubuntu]# 
[root@magna071 ubuntu]# reboot
Connection to magna071 closed by remote host.
Connection to magna071 closed.
vidushi@magna002:~$ ssh magna071
Warning: Permanently added 'magna071,10.8.128.71' (ECDSA) to the list of known hosts.
Last login: Wed Sep  6 16:38:05 2017 from 10.8.128.2
[ubuntu@magna071 ~]$ sudo -s
[root@magna071 ubuntu]# mount -t nfs -o vers=3 magna100:/ /home/ubuntu/mnt1
[root@magna071 ubuntu]# cd mnt1
[root@magna071 mnt1]# ls
bucket_1  bucket_version  bucket_version_1  bucket_version_new
[root@magna071 mnt1]#  
[root@magna071 mnt1]# rm -rf  bucket_version
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# rm -rf  bucket_version_new
rm: cannot remove ‘bucket_version_new’: Directory not empty
[root@magna071 mnt1]# 


Steps to Reproduce:
1. Configure Cluster with RGW and NFS
2. Create some directories with data using S3 and directory on NFS mount.
3. Perform RW IO to folders and buckets using different interface(S3 or kernel)
4. Delete all the files in the directory
5. Update the packages, unmount NFS, reboot the machine and remount it.
6. Try to remove the director

Actual results:
Deleting directories fails

Expected results:
Directory should be get deleted.

Additional info:

It is failing for some directories and passing for some.
Similar to bug https://bugzilla.redhat.com/show_bug.cgi?id=1422822
Comment 25 goocare 2020-08-05 04:01:48 UTC 
R-jsonlite-1.5-4.fc26 has been submitted as an update to Fedora 25 - https://8-ballpool.com/