Bug 1031030 - DHT + NFS mount : unlink of files using 'rm -rf *' failed with error 'Invalid argument'
DHT + NFS mount : unlink of files using 'rm -rf *' failed with error 'Invalid...
Status: CLOSED DEFERRED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
2.1
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Nithya Balachandran
storage-qa-internal@redhat.com
dht-rm-rf
:
Depends On:
Blocks: 1286044
  Show dependency treegraph
 
Reported: 2013-11-15 08:56 EST by Rachana Patel
Modified: 2015-11-27 05:13 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1286044 (view as bug list)
Environment:
Last Closed: 2015-11-27 05:13:17 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rachana Patel 2013-11-15 08:56:02 EST
Description of problem:
DHT volume was mounted using NFS and FUSE.
Created Directories and files on mount point.
From both mount point, executed 'rm -rf *' command and on NFS mount it failed for few files with error

e.g
rm: cannot remove `128/f27': Invalid argument
rm: cannot remove `128/f30': Invalid argument
rm: cannot remove `128/f32': Invalid argument
rm: cannot remove `128/f34': Invalid argument
rm: cannot remove `128/f35': Invalid argument
rm: cannot remove `128/f36': Invalid argument
rm: cannot remove `128/f40': Invalid argument
rm: cannot remove `128/f43': Invalid argument
rm: cannot remove `128/f49': Invalid argument
rm: cannot remove `128/f50': Invalid argument


nfs log snippet:-
[2013-11-15 09:26:00.571344] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.572630] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270
[2013-11-15 09:26:00.573564] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f40 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.573603] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270
[2013-11-15 09:26:00.573620] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: fd72ba23: /128/f40 => -1 (Invalid argument)
[2013-11-15 09:26:00.573634] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.574925] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757
[2013-11-15 09:26:00.576366] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f43 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757
[2013-11-15 09:26:00.576424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: ff72ba23: /128/f43 => -1 (Invalid argument)
[2013-11-15 09:26:00.576438] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: ff72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.577726] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367
[2013-11-15 09:26:00.578761] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f49 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.578805] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367
[2013-11-15 09:26:00.578823] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 173ba23: /128/f49 => -1 (Invalid argument)
[2013-11-15 09:26:00.578838] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: 173ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.580137] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097
[2013-11-15 09:26:00.581338] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f50 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.581394] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097
[2013-11-15 09:26:00.581424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 373ba23: /128/f50 => -1 (Invalid argument)


Version-Release number of selected component (if applicable):
============================================================= 
3.4.0.44rhs-1.el6rhs.x86_64



How reproducible:
always

Steps to Reproduce:
make sure all bricks are always up.
1.create DHT volume and mount it using NFS and FUSE
2. start creating data from mount points.
3. after creating few files and directories. execute 'rm -rf *' command from both mount point.
4. many times (not always), unlink on NFS mount fails with error 'Invalid argument'

Actual results:
It is failing with error 'Invalid argument'

Expected results:


Additional info:
Comment 3 Amar Tumballi 2013-12-02 05:12:41 EST
[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757

This is happening because of parallel 'rm -rf' operation on both the mounts.

IMO this is not a bug, but one can argue that i would expect a better error message, for which still keeping the bug open.
Comment 9 Susant Kumar Palai 2015-11-27 05:13:17 EST
Cloning this bug in 3.1. To be fixed in future release.

Note You need to log in before you can comment on or make changes to this bug.