Bug 1286200

Summary:	DHT:REBALANCE- statfs failures are seen during rebalance
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Susant Kumar Palai <spalai>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED WORKSFORME	QA Contact:	Prasad Desala <tdesala>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	kramdoss, nbalacha, rgowdapp, rhs-bugs, shmohan, smohan, spalai, storage-qa-internal, tdesala, vbellur
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:	dht-rebalance-file, dht-fops-while-rebal, dht-3.2.0-proposed
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1028287	Environment:
Last Closed:	2017-03-07 06:25:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1028287
Bug Blocks:

Comment 2 Nithya Balachandran 2016-01-11 10:04:39 UTC

*** Bug 1286207 has been marked as a duplicate of this bug. ***

Comment 3 krishnaram Karthick 2016-07-01 13:01:57 UTC

RCA:

dht_statfs_cbk should handle ENOENT/ESTALE errors just like dht_open_cbk/dht_stat_cbk as the file might've been moved out of the cached-subvol the client thinks.

regards,
Raghavendra

Comment 6 Prasad Desala 2017-02-01 07:42:53 UTC

This issue still exists with glusterfs version: 3.8.4-13.el7rhgs.x86_64.

Steps:
1) Created a ganesha cluster and created a distributed-replicate volume.
2) Enabled nfs-ganesha on the volume with mdcache settings.
3) Mounted the volume on multiple clients.
4) Created files and folders.
5) From mount point, issued rm -rf * and started removing bricks.

During remove-brick rebalance, statfs failures are seen in rebalance logs.

ERROR SNIPPET in rebalance logs:
================================
[2017-02-01 06:45:27.197342] E [MSGID: 109023] [dht-rebalance.c:779:__dht_check_free_space] 0-distrep-dht: failed to get statfs of /2/linux-4.4.1/drivers/net/wireless/ath/ath5k/ani.h on distrep-replicate-2 (No such file or directory)

Comment 7 Nithya Balachandran 2017-02-01 08:51:24 UTC

This is probably a valid scenario if the file is question was deleted just as the statfs was sent.

Do you see the issue with the steps listed in the description?

Comment 8 Prasad Desala 2017-03-06 17:04:13 UTC

This issue is not reproduced on glusterfs version 3.8.4-15.el7rhgs.x86_64. Followed the same steps as in the description on both FUSE and NFS but the error reported in this BZ is not seen.