Bug 1573083

Summary:	When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throws ESTALE errors
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Prasad Desala <tdesala>
Component:	distribute	Assignee:	Mohit Agrawal <moagrawa>
Status:	CLOSED WONTFIX	QA Contact:	Prasad Desala <tdesala>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.4	CC:	amukherj, ksubrahm, moagrawa, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	Cause: This type of scenario can come while no space is available on backend and client is trying to do file operation. Consequence: file operation is not success. Workaround (if any): No workaround is available Result:	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-19 03:48:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1503143

Description Prasad Desala 2018-04-30 07:33:43 UTC

Description of problem:
========================
When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throw ESTALE errors.

From mount point:
=================
Below dirs are created after the storage.reserve limits are reached.
d?????????? ? ?    ?            ?            ? dir_9
d?????????? ? ?    ?            ?            ? dir_4
d?????????? ? ?    ?            ?            ? dir_2
d?????????? ? ?    ?            ?            ? dir_10

On the backend the dir is created and the layout is missing on these dir.

Server-1:
# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-1=0x000000000000000000000000
trusted.afr.distrepx3-client-2=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick2/distrepx3-b2/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-6=0x000000000000000000000000
trusted.afr.distrepx3-client-7=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-15=0x000000000000000000000000
trusted.afr.distrepx3-client-17=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick5/distrepx3-b5/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-18=0x000000000000000000000000
trusted.afr.distrepx3-client-19=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-25=0x000000000000000000000000
trusted.afr.distrepx3-client-26=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-27=0x000000000000000000000000
trusted.afr.distrepx3-client-29=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-30=0x000000000000000000000000
trusted.afr.distrepx3-client-31=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-2:
# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-3:

# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-33=0x000000000000000000000000
trusted.afr.distrepx3-client-35=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-4:

# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick3/distrepx3-b3/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b


Version-Release number of selected component (if applicable):
3.12.2-8.el7rhgs.x86_64

How reproducible:
=================
always

Steps to Reproduce:
===================
1) Create a Distributed-Replicate volume and start it.
2) FUSE mount it on a client.
3) Set storage.reserve limit to 90%
4) Create files until the bricks reaches reserve limits.
5) Once the reserve limits are reached, file creation will fail with ENOSPC errors.
6) Now create 10 directories and dir creation will fail with ENOSPC errors.
7) Do ls <dirname> for the created dirs in step-6.

Actual results:
===============
dir creation fails with ENOSPC error but lookup on the dir throw ESTALE errors.

Expected results:
=================
lookup should not fail.