Description of problem: DHT volume was mounted using NFS and FUSE. Created Directories and files on mount point. From both mount point, executed 'rm -rf *' command and on NFS mount it failed for few files with error e.g rm: cannot remove `128/f27': Invalid argument rm: cannot remove `128/f30': Invalid argument rm: cannot remove `128/f32': Invalid argument rm: cannot remove `128/f34': Invalid argument rm: cannot remove `128/f35': Invalid argument rm: cannot remove `128/f36': Invalid argument rm: cannot remove `128/f40': Invalid argument rm: cannot remove `128/f43': Invalid argument rm: cannot remove `128/f49': Invalid argument rm: cannot remove `128/f50': Invalid argument nfs log snippet:- [2013-11-15 09:26:00.571344] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2013-11-15 09:26:00.572630] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270 [2013-11-15 09:26:00.573564] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f40 on dht-client-3: hashed subvolume cannot be found. [2013-11-15 09:26:00.573603] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270 [2013-11-15 09:26:00.573620] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: fd72ba23: /128/f40 => -1 (Invalid argument) [2013-11-15 09:26:00.573634] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2013-11-15 09:26:00.574925] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757 [2013-11-15 09:26:00.576366] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f43 on dht-client-3: hashed subvolume cannot be found. [2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757 [2013-11-15 09:26:00.576424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: ff72ba23: /128/f43 => -1 (Invalid argument) [2013-11-15 09:26:00.576438] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: ff72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2013-11-15 09:26:00.577726] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367 [2013-11-15 09:26:00.578761] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f49 on dht-client-3: hashed subvolume cannot be found. [2013-11-15 09:26:00.578805] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367 [2013-11-15 09:26:00.578823] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 173ba23: /128/f49 => -1 (Invalid argument) [2013-11-15 09:26:00.578838] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: 173ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2013-11-15 09:26:00.580137] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097 [2013-11-15 09:26:00.581338] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f50 on dht-client-3: hashed subvolume cannot be found. [2013-11-15 09:26:00.581394] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097 [2013-11-15 09:26:00.581424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 373ba23: /128/f50 => -1 (Invalid argument) Version-Release number of selected component (if applicable): ============================================================= 3.4.0.44rhs-1.el6rhs.x86_64 How reproducible: always Steps to Reproduce: make sure all bricks are always up. 1.create DHT volume and mount it using NFS and FUSE 2. start creating data from mount points. 3. after creating few files and directories. execute 'rm -rf *' command from both mount point. 4. many times (not always), unlink on NFS mount fails with error 'Invalid argument' Actual results: It is failing with error 'Invalid argument' Expected results: Additional info:
[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757 This is happening because of parallel 'rm -rf' operation on both the mounts. IMO this is not a bug, but one can argue that i would expect a better error message, for which still keeping the bug open.
Cloning this bug in 3.1. To be fixed in future release.