Bug 800803

Summary: All clients are marked fools due to "No space left on device"
Product: [Community] GlusterFS Reporter: Shwetha Panduranga <shwetha.h.panduranga>
Component: replicateAssignee: Jeff Darcy <jdarcy>
Status: CLOSED EOL QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: mainlineCC: bugs, gluster-bugs, jdarcy, pkarampu, rwheeler, spandura, vbellur, vinaraya
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.4.0qa5-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 853685 (view as bug list) Environment:
Last Closed: 2015-10-22 15:46:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 853685    

Description Shwetha Panduranga 2012-03-07 10:07:09 UTC
Description of problem:
If size of the files created on volume exceeds the space available on the volume all the clients are marked fools. 

Version-Release number of selected component (if applicable):
mainline

How reproducible:


Steps to Reproduce:
1.create a distribute-replicate volume. start the volume (each brick has 50G space available) 
2.create gluster,nfs mounts from client1
3.perform "dd if=/dev/zero of=gfsf1 bs=1M count=102400" from mount1
4.perform "dd if=/dev/zero of=nfsf1 bs=1M count=102400" from mount2
5.perform "dd if=/dev/urandom of=gfsf2 bs=1M count=102400" from mount3
6.perform "dd if=/dev/urandom of=nfsf2 bs=1M count=102400" from mount4
7.The file sizes created should exceed the space on the device.   

Actual results:
gluster volume info
 
Volume Name: datastore
Type: Distributed-Replicate
Volume ID: bc4bb820-400f-493e-bef7-ed09b87c8c91
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.2.35:/export1
Brick2: 192.168.2.36:/export1
Brick3: 192.168.2.35:/export2
Brick4: 192.168.2.36:/export2
Options Reconfigured:
diagnostics.brick-log-level: DEBUG
diagnostics.client-log-level: DEBUG

Brick1:-
---------
[03/07/12 - 20:33:10 root@APP-SERVER1 ~]# getfattr -R -m . -d -e hex /export1/*
getfattr: Removing leading '/' from absolute path names
# file: export1/nfsf1
trusted.afr.datastore-client-0=0x000001d10000000000000000
trusted.afr.datastore-client-1=0x000001d10000000000000000
trusted.gfid=0x4505225ade9d470290588082a5260ccb

Brick2:-
--------
[03/07/12 - 20:22:43 root@APP-SERVER2 glusterfs]# getfattr -m . -d -e hex /export1/*
getfattr: Removing leading '/' from absolute path names
# file: export1/nfsf1
trusted.afr.datastore-client-0=0x000001d10000000000000000
trusted.afr.datastore-client-1=0x000001d10000000000000000
trusted.gfid=0x4505225ade9d470290588082a5260ccb

Brick3:-
----------
[03/07/12 - 20:33:07 root@APP-SERVER1 ~]# getfattr -R -m . -d -e hex /export2/*
getfattr: Removing leading '/' from absolute path names
# file: export2/gfsf1
trusted.afr.datastore-client-2=0x0000000b0000000000000000
trusted.afr.datastore-client-3=0x0000000a0000000000000000
trusted.gfid=0x4ddef0724f4346d9b486a4a83ac649c6

# file: export2/gfsf2
trusted.afr.datastore-client-2=0x000000120000000000000000
trusted.afr.datastore-client-3=0x000000100000000000000000
trusted.gfid=0xfd5014406f67407f9abae7d3b97d7206

# file: export2/nfsf2
trusted.afr.datastore-client-2=0x00000d3e0000000000000000
trusted.afr.datastore-client-3=0x00000d3f0000000000000000
trusted.gfid=0x2f2a9de774d947e1830c8777ee4bbadf


Brick4:-
-------
[03/07/12 - 20:34:00 root@APP-SERVER2 glusterfs]# getfattr -m . -d -e hex /export2/*
getfattr: Removing leading '/' from absolute path names
# file: export2/gfsf1
trusted.afr.datastore-client-2=0x0000000b0000000000000000
trusted.afr.datastore-client-3=0x0000000a0000000000000000
trusted.gfid=0x4ddef0724f4346d9b486a4a83ac649c6

# file: export2/gfsf2
trusted.afr.datastore-client-2=0x000000120000000000000000
trusted.afr.datastore-client-3=0x000000100000000000000000
trusted.gfid=0xfd5014406f67407f9abae7d3b97d7206

# file: export2/nfsf2
trusted.afr.datastore-client-2=0x00000d3f0000000000000000
trusted.afr.datastore-client-3=0x00000d400000000000000000
trusted.gfid=0x2f2a9de774d947e1830c8777ee4bbadf


Expected results:

Additional info:

Comment 1 Shwetha Panduranga 2012-03-08 06:11:51 UTC
After marking the clients all-fools , lookup on files has various behaviors.

1) cat gfsf2/nfsf2 : Successful
2) ls -l gfsf1 : No such file or directory

[03/08/12 - 11:45:49 root@Shwetha-Laptop nfsc1]# ls -lh
ls: cannot access file10: Invalid argument
ls: cannot access gfsf1: Invalid argument
ls: cannot access gfsf2: Invalid argument
total 46G
-?????????? ? ?    ?       ?            ? file10
-?????????? ? ?    ?       ?            ? gfsf1
-?????????? ? ?    ?       ?            ? gfsf2
-rw-r--r--. 1 root root  41G Mar  8  2012 nfsf1
-rw-r--r--. 1 root root 4.9G Mar  7 23:33 nfsf2

[03/08/12 - 11:46:09 root@Shwetha-Laptop nfsc1]# ls -lh gfsf1
-rw-r--r--. 1 root root 41G Mar  8  2012 gfsf1
[03/08/12 - 11:46:24 root@Shwetha-Laptop nfsc1]# ls -lh
ls: cannot access file10: Invalid argument
ls: cannot access gfsf2: Invalid argument
total 86G
-?????????? ? ?    ?       ?            ? file10
-rw-r--r--. 1 root root  41G Mar  8  2012 gfsf1
-?????????? ? ?    ?       ?            ? gfsf2
-rw-r--r--. 1 root root  41G Mar  8  2012 nfsf1
-rw-r--r--. 1 root root 4.9G Mar  7 23:33 nfsf2

Comment 2 Jeff Darcy 2012-10-15 19:17:22 UTC
Shwetha, I'm unable to reproduce this on current master using the steps above.  Writes eventually fail with ENOSPEC (or sometimes EIO) but the changelogs are all clear and directory listings look normal.

Comment 3 Amar Tumballi 2013-01-11 07:50:31 UTC
as per comment #2, no specific fixes targetting this behavior, but please re-test initial scripts, and see if its happening.

Comment 4 Pranith Kumar K 2013-02-22 10:54:29 UTC
Please feel free to re-open the bug if it appears again.

Pranith.

Comment 6 spandura 2013-08-19 06:20:06 UTC
Discard comment 5 as it applies for the bug 853685 which is clone of this bug.

Comment 7 Kaleb KEITHLEY 2015-10-22 15:46:38 UTC
because of the large number of bugs filed against mainline version\ is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.