Description of problem: Due to stale link files in one of the replica pairs, rm -rf  on the mount point returned ENOTEMPTY. The sanity tests were running on the fuse mount point and one of the servers were brought down for a while. Self heal was triggered after bring up the server again. When I tried to delete the files, it returned ENOTEMPTY. Stat-prefetch was disabled and even after that deleting files returned the same error on the mount point.


Version-Release number of selected component (if applicable): 3.3.0qa22

  
Actual results:

When stat-prefetch was enabled-
[2012-02-16 02:32:11.782838] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readli
nk+0x230) [0x7fcaa95996c9] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readlink+0x28d) [0x7fca9ee9f29c] (-->/usr
/local/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readlink+0x217) [0x7fca9f0be09d]))) 0-anush-stat-prefetch: invalid argu
ment: inode
[2012-02-16 02:32:11.784779] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readv+
0x248) [0x7fcaa9597f7c] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readv+0x2b1) [0x7fca9eea1355] (-->/usr/local
/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readv+0x266) [0x7fca9f0c10e6]))) 0-anush-stat-prefetch: invalid argument: ino
de
[2012-02-16 02:32:11.785110] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readli
nk+0x230) [0x7fcaa95996c9] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readlink+0x28d) [0x7fca9ee9f29c] (-->/usr
/local/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readlink+0x217) [0x7fca9f0be09d]))) 0-anush-stat-prefetch: invalid argu
ment: inode
[2012-02-16 02:32:11.787231] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readv+
0x248) [0x7fcaa9597f7c] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readv+0x2b1) [0x7fca9eea1355] (-->/usr/local
/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readv+0x266) [0x7fca9f0c10e6]))) 0-anush-stat-prefetch: invalid argument: ino
de
[2012-02-16 02:32:11.803324] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readli
nk+0x230) [0x7fcaa95996c9] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readlink+0x28d) [0x7fca9ee9f29c] (-->/usr
/local/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readlink+0x217) [0x7fca9f0be09d]))) 0-anush-stat-prefetch: invalid argu
ment: inode
[2012-02-16 02:32:11.806388] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readv+
0x248) [0x7fcaa9597f7c] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readv+0x2b1) [0x7fca9eea1355] (-->/usr/local
/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readv+0x266) [0x7fca9f0c10e6]))) 0-anush-stat-prefetch: invalid argument: ino
de
[2012-02-16 02:32:11.807833] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readli
nk+0x230) [0x7fcaa95996c9] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readlink+0x28d) [0x7fca9ee9f29c] (-->/usr
/local/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readlink+0x217) [0x7fca9f0be09d]))) 0-anush-stat-prefetch: invalid argu
ment: inode
[2012-02-16 02:32:11.810305] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readv+
0x248) [0x7fcaa9597f7c] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readv+0x2b1) [0x7fca9eea1355] (-->/usr/local
/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readv+0x266) [0x7fca9f0c10e6]))) 0-anush-stat-prefetch: invalid argument: ino
de
[2012-02-16 02:32:11.810742] E [stat-prefetch.c:691:sp_remove_caches_from_all_fds_opened] (-->/usr/local/lib/libglusterfs.so.0(default_readli
nk+0x230) [0x7fcaa95996c9] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/debug/io-stats.so(io_stats_readlink+0x28d) [0x7fca9ee9f29c] (-->/usr
/local/lib/glusterfs/3.3.0qa22/xlator/performance/stat-prefetch.so(sp_readlink+0x217) [0x7fca9f0be09d]))) 0-anush-stat-prefetch: invalid argu
ment: inode


After disabling stat-prefetch-
[2012-02-16 19:52:42.700860] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:11:24.848635] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:11:24.910638] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:12:56.019936] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:12:56.041061] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:15:37.322858] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:15:37.350734] I [dict.c:339:dict_get] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/replicate.so(afr_readdirp_cbk+0xcb
8) [0x7fca9fd89382] (-->/usr/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_readdirp_cbk+0x8e) [0x7fca9fb59e34] (-->/us
r/local/lib/glusterfs/3.3.0qa22/xlator/cluster/distribute.so(dht_rmdir_is_subvol_empty+0xd2) [0x7fca9fb5991b]))) 1-dict: !this || key=trusted
.glusterfs.dht.linkto
[2012-02-16 20:17:48.956390] I [dht-common.c:1039:dht_lookup_linkfile_cbk] 1-anush-dht: lookup of /run23684/fstest_781dc696943ce44ad45d5c41b9
a45132/fstest_fd1655a444bd369b8401040bcddea5ba/fstest_ce5723b416cc62577509041ab6fc9e5e on anush-replicate-1 (following linkfile) failed (No s
uch file or directory)


Additional info:

After performing the stat on the file from the mount point on the stale file, I was able to delete the file successfully.

Type: Distributed-Replicate
Volume ID: 69f04722-5088-4424-9cee-8f7251879ce7
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.1.11.183:/newdata1
Brick2: 10.1.11.184:/newdata2
Brick3: 10.1.11.185:/newdata3
Brick4: 10.1.11.186:/newdata6
Options Reconfigured:
performance.stat-prefetch: off