Bug 1031030

Summary:	DHT + NFS mount : unlink of files using 'rm -rf *' failed with error 'Invalid argument'
Product:	Red Hat Gluster Storage	Reporter:	Rachana Patel <racpatel>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED DEFERRED	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.1	CC:	mzywusko, rwheeler, shaun, spalai, vagarwal, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:	dht-rm-rf
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1286044 (view as bug list)		Environment:
Last Closed:	2015-11-27 10:13:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1286044

Description Rachana Patel 2013-11-15 13:56:02 UTC

Description of problem:
DHT volume was mounted using NFS and FUSE.
Created Directories and files on mount point.
From both mount point, executed 'rm -rf *' command and on NFS mount it failed for few files with error

e.g
rm: cannot remove `128/f27': Invalid argument
rm: cannot remove `128/f30': Invalid argument
rm: cannot remove `128/f32': Invalid argument
rm: cannot remove `128/f34': Invalid argument
rm: cannot remove `128/f35': Invalid argument
rm: cannot remove `128/f36': Invalid argument
rm: cannot remove `128/f40': Invalid argument
rm: cannot remove `128/f43': Invalid argument
rm: cannot remove `128/f49': Invalid argument
rm: cannot remove `128/f50': Invalid argument


nfs log snippet:-
[2013-11-15 09:26:00.571344] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.572630] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270
[2013-11-15 09:26:00.573564] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f40 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.573603] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2084883270
[2013-11-15 09:26:00.573620] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: fd72ba23: /128/f40 => -1 (Invalid argument)
[2013-11-15 09:26:00.573634] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.574925] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757
[2013-11-15 09:26:00.576366] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f43 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757
[2013-11-15 09:26:00.576424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: ff72ba23: /128/f43 => -1 (Invalid argument)
[2013-11-15 09:26:00.576438] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: ff72ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.577726] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367
[2013-11-15 09:26:00.578761] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f49 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.578805] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 2094207367
[2013-11-15 09:26:00.578823] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 173ba23: /128/f49 => -1 (Invalid argument)
[2013-11-15 09:26:00.578838] W [nfs3-helpers.c:3395:nfs3_log_common_res] 0-nfs-nfsv3: XID: 173ba23, REMOVE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2013-11-15 09:26:00.580137] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097
[2013-11-15 09:26:00.581338] I [dht-common.c:869:dht_lookup_everywhere_done] 0-dht-dht: cannot create linkfile file for /128/f50 on dht-client-3: hashed subvolume cannot be found.
[2013-11-15 09:26:00.581394] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1378026097
[2013-11-15 09:26:00.581424] W [nfs3.c:3361:nfs3svc_remove_cbk] 0-nfs: 373ba23: /128/f50 => -1 (Invalid argument)


Version-Release number of selected component (if applicable):
============================================================= 
3.4.0.44rhs-1.el6rhs.x86_64



How reproducible:
always

Steps to Reproduce:
make sure all bricks are always up.
1.create DHT volume and mount it using NFS and FUSE
2. start creating data from mount points.
3. after creating few files and directories. execute 'rm -rf *' command from both mount point.
4. many times (not always), unlink on NFS mount fails with error 'Invalid argument'

Actual results:
It is failing with error 'Invalid argument'

Expected results:


Additional info:

Comment 3 Amar Tumballi 2013-12-02 10:12:41 UTC

[2013-11-15 09:26:00.576407] W [dht-layout.c:179:dht_layout_search] 0-dht-dht: no subvolume for hash (value) = 1919806757

This is happening because of parallel 'rm -rf' operation on both the mounts.

IMO this is not a bug, but one can argue that i would expect a better error message, for which still keeping the bug open.

Comment 9 Susant Kumar Palai 2015-11-27 10:13:17 UTC

Cloning this bug in 3.1. To be fixed in future release.