Bug 807246 - "rm -rf * " fails on the mount when there is no space left on the volume
"rm -rf * " fails on the mount when there is no space left on the volume
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: shishir gowda
: 794640 801332 (view as bug list)
Depends On:
Blocks: 817967
  Show dependency treegraph
Reported: 2012-03-27 07:06 EDT by Shwetha Panduranga
Modified: 2015-12-01 11:45 EST (History)
5 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-07-24 13:37:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
nfs server log (7.51 MB, text/x-log)
2012-03-27 07:06 EDT, Shwetha Panduranga
no flags Details
Brick3 Log (6.03 MB, text/x-log)
2012-03-27 07:08 EDT, Shwetha Panduranga
no flags Details
Fuse mount log file (445.68 KB, text/x-log)
2012-03-27 07:08 EDT, Shwetha Panduranga
no flags Details

  None (edit)
Description Shwetha Panduranga 2012-03-27 07:06:39 EDT
Created attachment 573020 [details]
nfs server log

Description of problem:
when the volume space was full, removal of files from the mount point failed because few directories on a brick (distribute-replicate volume) had the link files but the actual files for those link files were deleted. 

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1.create a distribute-replicate volume (2X2). start the volume
2.create fuse, nfs mounts. 
3.Start writes on the mount points. 
4.Bring down 2 bricks one from each replicate pair. 
5.continue the writes on mount point until the disk space is full
6.execute "rm -rf *" on the mount.
Actual results:
[03/27/12 - 20:20:41 root@APP-SERVER1 dir.2]# gluster volume info
Volume Name: dstore
Type: Distributed-Replicate
Volume ID: f934bb44-537a-429e-ab8a-3170f1949e54
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Options Reconfigured:
performance.stat-prefetch: off

output of execution of command "rm -rf *" on mount:
[03/27/12 - 19:22:03 root@APP-CLIENT1 nfsc1]# rm -rf *
rm: cannot remove `fuse1.3/dir.2': Directory not empty
rm: cannot remove `fuse1.3/dir.3': Directory not empty
rm: cannot remove `fuse2.1/dir.10': Directory not empty

Brick 3:-
[03/27/12 - 20:23:59 root@APP-SERVER1 dir.2]# ls /export2/dstore1/fuse1.3/dir.2 -a
.  ..

Brick 4:-
[03/27/12 - 20:23:44 root@APP-SERVER2 dir.2]# ls /export1/dstore1/fuse1.3/dir.2 -l
total 0
---------T 2 root root 0 Mar 27 19:10 file.1
---------T 2 root root 0 Mar 27 19:10 file.2
---------T 2 root root 0 Mar 27 19:10 file.3
---------T 2 root root 0 Mar 27 19:10 file.4
---------T 2 root root 0 Mar 27 19:10 file.5

[03/27/12 - 20:25:02 root@APP-SERVER2 dir.2]# getfattr -d -e hex -m. file.*
# file: file.1

# file: file.2

# file: file.3

# file: file.4

# file: file.5
Comment 1 Shwetha Panduranga 2012-03-27 07:08:00 EDT
Created attachment 573021 [details]
Brick3 Log
Comment 2 Shwetha Panduranga 2012-03-27 07:08:48 EDT
Created attachment 573022 [details]
Fuse mount log file
Comment 3 shishir gowda 2012-03-27 07:44:02 EDT
The issue seems to be this
1. A link file was created as hashed subvol was full
2. All subvols were full, which led the creation of the file to fail
3. Following readdirp do not show these dangling link files(as we ignore link files).
4. rm fails with ENOTEMTPY
Comment 4 Anand Avati 2012-03-29 10:45:30 EDT
CHANGE: http://review.gluster.com/3035 (cluster/dht: send dict with linkto key for lookup in rmdir_readdirp) merged in master by Vijay Bellur (vijay@gluster.com)
Comment 5 shishir gowda 2012-03-30 02:36:29 EDT
*** Bug 794640 has been marked as a duplicate of this bug. ***
Comment 6 shishir gowda 2012-03-30 04:48:30 EDT
*** Bug 801332 has been marked as a duplicate of this bug. ***
Comment 7 Shwetha Panduranga 2012-04-12 01:58:31 EDT
Verified on 3.3.0qa34.
Comment 8 Christian Horn 2013-03-21 05:27:59 EDT
I suspect this bz is already CLOSED ERRATA and the patches went into the packages?

The current builds are not even mentioning 3.3.0qa34 any more:
# rpm -q --changelog glusterfs- | tail -2
- Updated to gluster 3.3 qa40 build

I am wondering weather we hit the same issue in bz922792 .

Note You need to log in before you can comment on or make changes to this bug.