Bug 1573083 - When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throws ESTALE errors
Summary: When storage.reserve limits are reached and directory is created, dir creatio...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Mohit Agrawal
QA Contact: Prasad Desala
URL:
Whiteboard:
Depends On:
Blocks: 1503143
TreeView+ depends on / blocked
 
Reported: 2018-04-30 07:33 UTC by Prasad Desala
Modified: 2019-11-19 03:48 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: This type of scenario can come while no space is available on backend and client is trying to do file operation. Consequence: file operation is not success. Workaround (if any): No workaround is available Result:
Clone Of:
Environment:
Last Closed: 2019-11-19 03:48:38 UTC
Embargoed:

Attachments (Terms of Use)

Description Prasad Desala 2018-04-30 07:33:43 UTC 
Description of problem:
========================
When storage.reserve limits are reached and directory is created, dir creation fails with ENOSPC error but lookup on the dir throw ESTALE errors.

From mount point:
=================
Below dirs are created after the storage.reserve limits are reached.
d?????????? ? ?    ?            ?            ? dir_9
d?????????? ? ?    ?            ?            ? dir_4
d?????????? ? ?    ?            ?            ? dir_2
d?????????? ? ?    ?            ?            ? dir_10

On the backend the dir is created and the layout is missing on these dir.

Server-1:
# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-1=0x000000000000000000000000
trusted.afr.distrepx3-client-2=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick2/distrepx3-b2/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-6=0x000000000000000000000000
trusted.afr.distrepx3-client-7=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-15=0x000000000000000000000000
trusted.afr.distrepx3-client-17=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick5/distrepx3-b5/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-18=0x000000000000000000000000
trusted.afr.distrepx3-client-19=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-25=0x000000000000000000000000
trusted.afr.distrepx3-client-26=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-27=0x000000000000000000000000
trusted.afr.distrepx3-client-29=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-30=0x000000000000000000000000
trusted.afr.distrepx3-client-31=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-2:
# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-3:

# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.distrepx3-client-33=0x000000000000000000000000
trusted.afr.distrepx3-client-35=0x000000000000000000000000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

Server-4:

# file: bricks/brick0/distrepx3-b0/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b
trusted.glusterfs.dht.mds=0x00000000

# file: bricks/brick1/distrepx3-b1/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick3/distrepx3-b3/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick4/distrepx3-b4/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick6/distrepx3-b6/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick7/distrepx3-b7/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b

# file: bricks/brick8/distrepx3-b8/dir_9
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x0016f0ee4f0a482dba210c416cd9cb4b


Version-Release number of selected component (if applicable):
3.12.2-8.el7rhgs.x86_64

How reproducible:
=================
always

Steps to Reproduce:
===================
1) Create a Distributed-Replicate volume and start it.
2) FUSE mount it on a client.
3) Set storage.reserve limit to 90%
4) Create files until the bricks reaches reserve limits.
5) Once the reserve limits are reached, file creation will fail with ENOSPC errors.
6) Now create 10 directories and dir creation will fail with ENOSPC errors.
7) Do ls <dirname> for the created dirs in step-6.

Actual results:
===============
dir creation fails with ENOSPC error but lookup on the dir throw ESTALE errors.

Expected results:
=================
lookup should not fail.
Note You need to log in before you can comment on or make changes to this bug.