Bug 1110457

Summary:	DHT + NFS :- Directory creation failed with 'Invalid argument' when one sub-volume is down
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rachana Patel <racpatel>
Component:	distribute	Assignee:	Satish Mohan <smohan>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.0	CC:	mzywusko, nbalacha, rgowdapp, smohan, tdesala
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:	dht-fixed
Fixed In Version:	3.7.9-10	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-19 06:29:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rachana Patel 2014-06-17 17:46:10 UTC

Description of problem:
=======================
On Distributed volume when one sub-volume was down , directory creation failed with error 'Invalid argument'


Version-Release number of selected component (if applicable):
=============================================================
3.6.0.18-1.el6rhs.x86_64


How reproducible:
================
not always


Steps to Reproduce:
===================
1.create and mount distributed volume(NFS)
2. bring one or more sub-volume down by killing brick process.
3. add 2 more brick to volume
4. start I/O on NFs mount(dir and file cration)


Actual results:
==============
Directory creation  failed with error 'Invalid argument' as below

cannot create directory `etc1': Invalid argument



Expected results:
=================
If hashed sub-volume is down then on NFs mount it should fail with "I/O error".
not with 'Invalid argument'


Additional info:
log snippet
[2014-06-17 08:18:48.220398] W [dht-layout.c:178:dht_layout_search] 0-test1-dht: no subvolume for hash (value) = 2326726385
[2014-06-17 08:18:48.220428] W [nfs3.c:2722:nfs3svc_mkdir_cbk] 0-nfs: d2438e2b: /1/etc1 => -1 (Invalid argument)
[2014-06-17 08:18:48.220450] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: d2438e2b, MKDIR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2014-06-17 08:18:48.224624] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-test1-client-4: remote operation failed: Stale file handle. Path: /1/2
[2014-06-17 08:18:48.224788] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-test1-client-3: remote operation failed: Stale file handle. Path: /1/2
[2014-06-17 08:18:48.225446] W [client-rpc-fops.c:1021:client3_3_setxattr_cbk] 0-test1-client-4: remote operation failed: Stale file handle
[2014-06-17 08:18:48.225522] W [client-rpc-fops.c:1021:client3_3_setxattr_cbk] 0-test1-client-3: remote operation failed: Stale file handle

Comment 6 Mike McCune 2016-03-28 23:31:34 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 7 Raghavendra G 2016-06-22 10:10:09 UTC

Fix has gone into downstream as part of rebase of 3.7 branch. Patch can be found at http://review.gluster.org/13351

Comment 9 Prasad Desala 2016-08-19 05:31:44 UTC

This issue is not seen on glusterfs build: 3.7.9-10.el7rhgs.x86_64.

Here are the steps that were performed,

1. Created a distributed replica volume and NFS mounted the volume to a client.
2. Brought down few sub-vols by killing brick process.
3. Added few more bricks to the volume.
4. Started I/O on NFS mount(dir and file creation).

During dir/file creation if the hashed subvol is not found, error "Input/output" is seen which is the fix.
mkdir: cannot create directory ‘d70’: Input/output error
dd: failed to open ‘file_17’: Input/output error

Also, this was tested with FUSE mount and appropriate error is seen.

Hence, marking this bug as Verified.

Comment 10 Nithya Balachandran 2016-08-19 06:29:05 UTC

Thanks Prasad.

As per comment#9, I am Closing this BZ.