Created attachment 573020 [details] nfs server log Description of problem: when the volume space was full, removal of files from the mount point failed because few directories on a brick (distribute-replicate volume) had the link files but the actual files for those link files were deleted. Version-Release number of selected component (if applicable): mainline Steps to Reproduce: 1.create a distribute-replicate volume (2X2). start the volume 2.create fuse, nfs mounts. 3.Start writes on the mount points. 4.Bring down 2 bricks one from each replicate pair. 5.continue the writes on mount point until the disk space is full 6.execute "rm -rf *" on the mount. Actual results: [03/27/12 - 20:20:41 root@APP-SERVER1 dir.2]# gluster volume info Volume Name: dstore Type: Distributed-Replicate Volume ID: f934bb44-537a-429e-ab8a-3170f1949e54 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 192.168.2.35:/export1/dstore1 Brick2: 192.168.2.36:/export1/dstore1 Brick3: 192.168.2.35:/export2/dstore1 Brick4: 192.168.2.36:/export2/dstore1 Options Reconfigured: performance.stat-prefetch: off nfs.rpc-auth-allow: 192.168.2.34 auth.allow: 192.168.2.34 output of execution of command "rm -rf *" on mount: ---------------------------------------------------- [03/27/12 - 19:22:03 root@APP-CLIENT1 nfsc1]# rm -rf * rm: cannot remove `fuse1.3/dir.2': Directory not empty rm: cannot remove `fuse1.3/dir.3': Directory not empty rm: cannot remove `fuse2.1/dir.10': Directory not empty Brick 3:- -------- [03/27/12 - 20:23:59 root@APP-SERVER1 dir.2]# ls /export2/dstore1/fuse1.3/dir.2 -a . .. Brick 4:- -------- [03/27/12 - 20:23:44 root@APP-SERVER2 dir.2]# ls /export1/dstore1/fuse1.3/dir.2 -l total 0 ---------T 2 root root 0 Mar 27 19:10 file.1 ---------T 2 root root 0 Mar 27 19:10 file.2 ---------T 2 root root 0 Mar 27 19:10 file.3 ---------T 2 root root 0 Mar 27 19:10 file.4 ---------T 2 root root 0 Mar 27 19:10 file.5 [03/27/12 - 20:25:02 root@APP-SERVER2 dir.2]# getfattr -d -e hex -m. file.* # file: file.1 trusted.gfid=0x13d996b62d40409c81fa3da5ff83edf4 trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100 # file: file.2 trusted.gfid=0x1602edf3908a4cab96b02ae6c83bbf19 trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100 # file: file.3 trusted.gfid=0xadda3b8a5f5c48c5a745a51672662bcf trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100 # file: file.4 trusted.gfid=0x2c9ee186c1d54dcfb76fc3ac9ee333db trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100 # file: file.5 trusted.gfid=0x54d99802a9a440f0af7f7db3c795c711 trusted.glusterfs.dht.linkto=0x6473746f72652d7265706c69636174652d3100
Created attachment 573021 [details] Brick3 Log
Created attachment 573022 [details] Fuse mount log file
The issue seems to be this 1. A link file was created as hashed subvol was full 2. All subvols were full, which led the creation of the file to fail 3. Following readdirp do not show these dangling link files(as we ignore link files). 4. rm fails with ENOTEMTPY
CHANGE: http://review.gluster.com/3035 (cluster/dht: send dict with linkto key for lookup in rmdir_readdirp) merged in master by Vijay Bellur (vijay)
*** Bug 794640 has been marked as a duplicate of this bug. ***
*** Bug 801332 has been marked as a duplicate of this bug. ***
Verified on 3.3.0qa34.
I suspect this bz is already CLOSED ERRATA and the patches went into the packages? The current builds are not even mentioning 3.3.0qa34 any more: # rpm -q --changelog glusterfs-3.3.0.5rhs-37.el6rhs.x86_64 | tail -2 - Updated to gluster 3.3 qa40 build I am wondering weather we hit the same issue in bz922792 .