Bug 1489301 - [RGW:NFS]: Directories not getting removed from NFS mount specifically when versioning is enabled
Summary: [RGW:NFS]: Directories not getting removed from NFS mount specifically when v...
Status: ON_QA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 3.0
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: rc
: 4.0
Assignee: Matt Benjamin (redhat)
QA Contact: vidushi
Bara Ancincova
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1494421
TreeView+ depends on / blocked
 
Reported: 2017-09-07 07:31 UTC by vidushi
Modified: 2019-05-31 02:25 UTC (History)
9 users (show)

(edit)
.It is not possible to remove directories stored on S3 versioned buckets by using `rm`

The mechanism that is used to check for non-empty directories prior to unlinking them works incorrectly in combination with the Ceph Object Gateway Simple Storage Service (S3) versioned buckets. As a consequence, directory trees on versioned buckets cannot be recursively removed with a command such as `rm -rf`. To work around this problem, remove any objects in versioned buckets by using the S3 interface.
Clone Of:
(edit)
Last Closed:


Attachments (Terms of Use)

Description vidushi 2017-09-07 07:31:05 UTC
Description of problem:

Tried to remove a empty directory from NFS mount but it failed with "rm: cannot remove ‘bucket_version/’: Directory not empty" error.

Created some buckets with data using s3 some days before, and removed all data and there was only empty directory. 


Version-Release number of selected component (if applicable):

ceph version: 12.2.0-1.el7cp
nfs-ganesha version: 
nfs-ganesha-2.5.2-1.el7cp.x86_64
nfs-ganesha-rgw-2.5.2-1.el7cp.x86_64


How reproducible:
1/1


[root@magna071 mnt1]# cd bucket_version/
[root@magna071 bucket_version]# ls
[root@magna071 bucket_version]# ll -h
total 0
[root@magna071 bucket_version]# cd .. 
[root@magna071 mnt1]# ls
bucket1  bucket_version

[root@magna071 mnt1]# date
Wed Sep  6 07:01:12 UTC 2017
[root@magna071 mnt1]# rm -rf *
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# ls
bucket_version
[root@magna071 mnt1]# rm -rf *
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# 


After updating packages, umounting, rebooting and remounting also faced the same error.


[root@magna071 ubuntu]# umount  -l mnt1
[root@magna071 ubuntu]# 
[root@magna071 ubuntu]# reboot
Connection to magna071 closed by remote host.
Connection to magna071 closed.
vidushi@magna002:~$ ssh magna071
Warning: Permanently added 'magna071,10.8.128.71' (ECDSA) to the list of known hosts.
Last login: Wed Sep  6 16:38:05 2017 from 10.8.128.2
[ubuntu@magna071 ~]$ sudo -s
[root@magna071 ubuntu]# mount -t nfs -o vers=3 magna100:/ /home/ubuntu/mnt1
[root@magna071 ubuntu]# cd mnt1
[root@magna071 mnt1]# ls
bucket_1  bucket_version  bucket_version_1  bucket_version_new
[root@magna071 mnt1]#  
[root@magna071 mnt1]# rm -rf  bucket_version
rm: cannot remove ‘bucket_version’: Directory not empty
[root@magna071 mnt1]# rm -rf  bucket_version_new
rm: cannot remove ‘bucket_version_new’: Directory not empty
[root@magna071 mnt1]# 


Steps to Reproduce:
1. Configure Cluster with RGW and NFS
2. Create some directories with data using S3 and directory on NFS mount.
3. Perform RW IO to folders and buckets using different interface(S3 or kernel)
4. Delete all the files in the directory
5. Update the packages, unmount NFS, reboot the machine and remount it.
6. Try to remove the director

Actual results:
Deleting directories fails

Expected results:
Directory should be get deleted.

Additional info:

It is failing for some directories and passing for some.
Similar to bug https://bugzilla.redhat.com/show_bug.cgi?id=1422822


Note You need to log in before you can comment on or make changes to this bug.