Bug 1031030 - DHT + NFS mount : unlink of files using 'rm -rf *' failed with error 'Invalid argument'
Summary: DHT + NFS mount : unlink of files using 'rm -rf *' failed with error 'Invalid...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: 2.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Nithya Balachandran
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard: dht-rm-rf
Depends On:
Blocks: 1286044
TreeView+ depends on / blocked
 
Reported: 2013-11-15 13:56 UTC by Rachana Patel
Modified: 2015-11-27 10:13 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1286044 (view as bug list)
Environment:
Last Closed: 2015-11-27 10:13:17 UTC
Embargoed:


Attachments (Terms of Use)

Description Rachana Patel 2013-11-15 13:56:02 UTC
Description of problem:
DHT volume was mounted using NFS and FUSE.
Created Directories and files on mount point.
From both mount point, executed 'rm -rf *' command and on NFS mount it failed for few files with error

e.g
rm: cannot remove `128/f27': Invalid argument
rm: cannot remove `128/f30': Invalid argument
rm: cannot remove `128/f32': Invalid argument
rm: cannot remove `128/f34': Invalid argument
rm: cannot remove `128/f35': Invalid argument
rm: cannot remove `128/f36': Invalid argument
rm: cannot remove `128/f40': Invalid argument
rm: cannot remove `128/f43': Invalid argument
rm: cannot remove `128/f49': Invalid argument
rm: cannot remove `128/f50': Invalid argument


nfs log snippet:-
[2013-11-15 09:26:00.571344] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.572630] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270
[2013-11-15 09:26:00.573564] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f40 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.573603] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270
[2013-11-15 09:26:00.573620] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: fd72ba23: /128/f40 => -1 (Invalid argument)
[2013-11-15 09:26:00.573634] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.574925] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757
[2013-11-15 09:26:00.576366] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f43 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757
[2013-11-15 09:26:00.576424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: ff72ba23: /128/f43 => -1 (Invalid argument)
[2013-11-15 09:26:00.576438] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: ff72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.577726] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367
[2013-11-15 09:26:00.578761] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f49 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.578805] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367
[2013-11-15 09:26:00.578823] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 173ba23: /128/f49 => -1 (Invalid argument)
[2013-11-15 09:26:00.578838] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: 173ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.580137] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097
[2013-11-15 09:26:00.581338] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f50 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.581394] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097
[2013-11-15 09:26:00.581424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 373ba23: /128/f50 => -1 (Invalid argument)


Version-Release number of selected component (if applicable):
============================================================= 
3.4.0.44rhs-1.el6rhs.x86_64



How reproducible:
always

Steps to Reproduce:
make sure all bricks are always up.
1.create DHT volume and mount it using NFS and FUSE
2. start creating data from mount points.
3. after creating few files and directories. execute 'rm -rf *' command from both mount point.
4. many times (not always), unlink on NFS mount fails with error 'Invalid argument'

Actual results:
It is failing with error 'Invalid argument'

Expected results:


Additional info:

Comment 3 Amar Tumballi 2013-12-02 10:12:41 UTC
[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757

This is happening because of parallel 'rm -rf' operation on both the mounts.

IMO this is not a bug, but one can argue that i would expect a better error message, for which still keeping the bug open.

Comment 9 Susant Kumar Palai 2015-11-27 10:13:17 UTC
Cloning this bug in 3.1. To be fixed in future release.


Note You need to log in before you can comment on or make changes to this bug.