Bug 807246 - "rm -rf * " fails on the mount when there is no space left on the volume
"rm -rf * " fails on the mount when there is no space left on the volume
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: shishir gowda
:
: 794640 801332 (view as bug list)
Depends On:
Blocks: 817967
  Show dependency treegraph
 
Reported: 2012-03-27 07:06 EDT by Shwetha Panduranga
Modified: 2015-12-01 11:45 EST (History)
5 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:37:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
nfs server log (7.51 MB, text/x-log)
2012-03-27 07:06 EDT, Shwetha Panduranga
no flags Details
Brick3 Log (6.03 MB, text/x-log)
2012-03-27 07:08 EDT, Shwetha Panduranga
no flags Details
Fuse mount log file (445.68 KB, text/x-log)
2012-03-27 07:08 EDT, Shwetha Panduranga
no flags Details

  None (edit)
Description Shwetha Panduranga 2012-03-27 07:06:39 EDT
Created attachment 573020 [details]
nfs server log

Description of problem:
when the volume space was full, removal of files from the mount point failed because few directories on a brick (distribute-replicate volume) had the link files but the actual files for those link files were deleted. 

Version-Release number of selected component (if applicable):
mainline

Steps to Reproduce:
1.create a distribute-replicate volume (2X2). start the volume
2.create fuse, nfs mounts. 
3.Start writes on the mount points. 
4.Bring down 2 bricks one from each replicate pair. 
5.continue the writes on mount point until the disk space is full
6.execute "rm -rf *" on the mount.
  
Actual results:
[03/27/12 - 20:20:41 root@APP-SERVER1 dir.2]# gluster volume info
 
Volume Name: dstore
Type: Distributed-Replicate
Volume ID: f934bb44-537a-429e-ab8a-3170f1949e54
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.2.35:/export1/dstore1
Brick2: 192.168.2.36:/export1/dstore1
Brick3: 192.168.2.35:/export2/dstore1
Brick4: 192.168.2.36:/export2/dstore1
Options Reconfigured:
performance.stat-prefetch: off
nfs.rpc-auth-allow: 192.168.2.34
auth.allow: 192.168.2.34


output of execution of command "rm -rf *" on mount:
----------------------------------------------------
[03/27/12 - 19:22:03 root@APP-CLIENT1 nfsc1]# rm -rf *
rm: cannot remove `fuse1.3/dir.2': Directory not empty
rm: cannot remove `fuse1.3/dir.3': Directory not empty
rm: cannot remove `fuse2.1/dir.10': Directory not empty

Brick 3:-
--------
[03/27/12 - 20:23:59 root@APP-SERVER1 dir.2]# ls /export2/dstore1/fuse1.3/dir.2 -a
.  ..

Brick 4:-
--------
[03/27/12 - 20:23:44 root@APP-SERVER2 dir.2]# ls /export1/dstore1/fuse1.3/dir.2 -l
total 0
---------T 2 root root 0 Mar 27 19:10 file.1
---------T 2 root root 0 Mar 27 19:10 file.2
---------T 2 root root 0 Mar 27 19:10 file.3
---------T 2 root root 0 Mar 27 19:10 file.4
---------T 2 root root 0 Mar 27 19:10 file.5

[03/27/12 - 20:25:02 root@APP-SERVER2 dir.2]# getfattr -d -e hex -m. file.*
# file: file.1
trusted.gfid=0x13d996b62d40409c81fa3da5ff83edf4
trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100

# file: file.2
trusted.gfid=0x1602edf3908a4cab96b02ae6c83bbf19
trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100

# file: file.3
trusted.gfid=0xadda3b8a5f5c48c5a745a51672662bcf
trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100

# file: file.4
trusted.gfid=0x2c9ee186c1d54dcfb76fc3ac9ee333db
trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100

# file: file.5
trusted.gfid=0x54d99802a9a440f0af7f7db3c795c711
trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100
Comment 1 Shwetha Panduranga 2012-03-27 07:08:00 EDT
Created attachment 573021 [details]
Brick3 Log
Comment 2 Shwetha Panduranga 2012-03-27 07:08:48 EDT
Created attachment 573022 [details]
Fuse mount log file
Comment 3 shishir gowda 2012-03-27 07:44:02 EDT
The issue seems to be this
1. A link file was created as hashed subvol was full
2. All subvols were full, which led the creation of the file to fail
3. Following readdirp do not show these dangling link files(as we ignore link files).
4. rm fails with ENOTEMTPY
Comment 4 Anand Avati 2012-03-29 10:45:30 EDT
CHANGE: http://review.gluster.com/3035 (cluster/dht: send dict with linkto key for lookup in rmdir_readdirp) merged in master by Vijay Bellur (vijay@gluster.com)
Comment 5 shishir gowda 2012-03-30 02:36:29 EDT
*** Bug 794640 has been marked as a duplicate of this bug. ***
Comment 6 shishir gowda 2012-03-30 04:48:30 EDT
*** Bug 801332 has been marked as a duplicate of this bug. ***
Comment 7 Shwetha Panduranga 2012-04-12 01:58:31 EDT
Verified on 3.3.0qa34.
Comment 8 Christian Horn 2013-03-21 05:27:59 EDT
I suspect this bz is already CLOSED ERRATA and the patches went into the packages?

The current builds are not even mentioning 3.3.0qa34 any more:
# rpm -q --changelog glusterfs-3.3.0.5rhs-37.el6rhs.x86_64 | tail -2
- Updated to gluster 3.3 qa40 build

I am wondering weather we hit the same issue in bz922792 .

Note You need to log in before you can comment on or make changes to this bug.