Bug 1573083 – When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throws ESTALE errors

Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.

Bug 1573083 - When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throws ESTALE errors

Summary:	When storage.reserve limits are reached and directory is created, dir creatio...

Status:	NEW

Aliases:	None

Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	distribute (Show other bugs)
Sub Component:
Version:	3.4
Hardware:	Unspecified Unspecified

Priority	unspecified Severity medium
Target Milestone:	---
Target Release:	---
Assigned To:	Mohit Agrawal
QA Contact:	Prasad Desala
Docs Contact:

URL:
Whiteboard:
Keywords:	ZStream

Depends On:
Blocks:	1503143
	Show dependency tree / graph

Reported:	2018-04-30 03:33 EDT by Prasad Desala
Modified:	2018-08-28 01:48 EDT (History)
CC List:	8 users (show)

See Also:
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	Cause: This type of scenario can come while no space is available on backend and client is trying to do file operation. Consequence: file operation is not success. Workaround (if any): No workaround is available Result:
Story Points:	---
Clone Of:
Environment:
Last Closed:
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Groups: None (edit)

Description Prasad Desala 2018-04-30 03:33:43 EDT

Description of problem:
========================
When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throw ESTALE errors.

From mount point:
=================
Below dirs are created after the storage.reserve limits are reached.
d?????????? ? ?    ?            ?            ? dir_9
d?????????? ? ?    ?            ?            ? dir_4
d?????????? ? ?    ?            ?            ? dir_2
d?????????? ? ?    ?            ?            ? dir_10

On the backend the dir is created and the layout is missing on these dir.

Server-1:
# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-1=0x000000000000000000000000
trusted.afr.distrepx3-client-2=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick2/distrepx3-b2/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-6=0x000000000000000000000000
trusted.afr.distrepx3-client-7=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-15=0x000000000000000000000000
trusted.afr.distrepx3-client-17=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick5/distrepx3-b5/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-18=0x000000000000000000000000
trusted.afr.distrepx3-client-19=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-25=0x000000000000000000000000
trusted.afr.distrepx3-client-26=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-27=0x000000000000000000000000
trusted.afr.distrepx3-client-29=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-30=0x000000000000000000000000
trusted.afr.distrepx3-client-31=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-2:
# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-3:

# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-33=0x000000000000000000000000
trusted.afr.distrepx3-client-35=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-4:

# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick3/distrepx3-b3/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b


Version-Release number of selected component (if applicable):
3.12.2-8.el7rhgs.x86_64

How reproducible:
=================
always

Steps to Reproduce:
===================
1) Create a Distributed-Replicate volume and start it.
2) FUSE mount it on a client.
3) Set storage.reserve limit to 90%
4) Create files until the bricks reaches reserve limits.
5) Once the reserve limits are reached, file creation will fail with ENOSPC errors.
6) Now create 10 directories and dir creation will fail with ENOSPC errors.
7) Do ls <dirname> for the created dirs in step-6.

Actual results:
===============
dir creation fails with ENOSPC error but lookup on the dir throw ESTALE errors.

Expected results:
=================
lookup should not fail.

Note You need to log in before you can comment on or make changes to this bug.