Bug 1287440 - When GlusterFS mount volume is full, create a new file and read the file, the error code shows No such file or directory, but the file indeed is exist
When GlusterFS mount volume is full, create a new file and read the file, the...
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core (Show other bugs)
unspecified
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
Anoop
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-02 01:13 EST by coyang
Modified: 2017-03-25 12:26 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description coyang 2015-12-02 01:13:09 EST
Description of problem:


Version-Release number of selected component (if applicable):
# glusterfs -V
glusterfs 3.6.2 built on Sep  1 2015 13:55:09


How reproducible:
Pre-condition: 
on client node, /mnt/export is mount by glusterfs
# df |grep export
169.254.0.90:/export      4782720   30464   4473728   1% /mnt/export
[root@CFPU-0(matrix) /home/_rcpadmin]
# mount |grep export
169.254.0.90:/export on /mnt/export type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

1. use dd make large file in /mnt/export which result in /mnt/export full
[root@CFPU-0(matrix) /home/_rcpadmin]
# dd if=/dev/zero of=/mnt/export/testfull bs=10M count=1000
dd: error writing ‘/mnt/export/testfull’: No space left on device
dd: closing output file ‘/mnt/export/testfull’: No space left on device
[root@CFPU-0(matrix) /home/_rcpadmin]
# ls /mnt/export/
fsconfigure.pid  symptomreport	test2.log  testfull  test.log  trace
[root@CFPU-0(matrix) /home/_rcpadmin]
# df /mnt/export/
Filesystem           1K-blocks    Used Available Use% Mounted on
169.254.0.90:/export   4782720 4766336         0 100% /mnt/export

2. tail log to a new log named testlog.txt,
[root@CFPU-0(matrix) /home/_rcpadmin]
# tail -n 100 /var/log/syslog >/mnt/export/testlog.txt
tail: write error: No space left on device
[root@CFPU-0(matrix) /home/_rcpadmin]

3. cat the file return No such file or directory, but ls shows the file is exist

[root@CFPU-0(matrix) /home/_rcpadmin]
# ls -al /mnt/export/
total 4735934
drwxrws---  5 root _nokfsuifileshare       4096 Dec  2 13:55 .
drwxr-xr-x 17 root root                    4096 Nov 30 16:07 ..
-rwxr-xr-x  1 root _nokfsuifileshare         17 Dec  1 21:08 fsconfigure.pid
drwxr-x---  2 root _nokfsuitracelog        4096 Nov 30 16:03 symptomreport
-rw-r--r--  1 root _nokfsuifileshare       1861 Dec  1 20:43 test2.log
-rw-r--r--  1 root _nokfsuifileshare 4849561600 Dec  2 13:47 testfull
-rw-r--r--  1 root _nokfsuifileshare      14906 Dec  1 20:43 test.log
-rw-r--r--  0 root _nokfsuifileshare          0 Dec  2 13:55 testlog.txt
drwxr-s---  2 root _nokfsuitracelog        4096 Nov 30 16:03 trace
[root@CFPU-0(matrix) /home/_rcpadmin]
# cat /mnt/export/testlog.txt 
cat: /mnt/export/testlog.txt: No such file or directory

4. ssh to storage node, the file is exist on /mnt/bricks/export/, cat can be success execute on local ext4 file system
# ssh _rcpadmin@169.254.0.90
Password: 
[root@SN-0(matrix) /home/_rcpadmin]
# ls -al /mnt/bricks/export/
brick/      lost+found/ 
[root@SN-0(matrix) /home/_rcpadmin]        
[root@SN-0(matrix) /home/_rcpadmin]
# ls -al /mnt/bricks/export/brick/
total 4735968
drwxrws---   5 root _nokfsuifileshare       4096 Dec  2 13:55 .
drwxr-xr-x   4 root root                    4096 Nov 30 15:59 ..
-rwxr-xr-x   2 root _nokfsuifileshare         17 Dec  1 21:08 fsconfigure.pid
drw------- 261 root root                    4096 Dec  1 11:45 .glusterfs
drwxr-x---   2 root _nokfsuitracelog        4096 Nov 30 16:03 symptomreport
-rw-r--r--   2 root _nokfsuifileshare       1861 Dec  1 20:43 test2.log
-rw-r--r--   2 root _nokfsuifileshare 4849561600 Dec  2 13:47 testfull
-rw-r--r--   2 root _nokfsuifileshare      14906 Dec  1 20:43 test.log
-rw-r--r--   1 root _nokfsuifileshare          0 Dec  2 13:55 testlog.txt
drwxr-s---   2 root _nokfsuitracelog        4096 Nov 30 16:03 trace


# cat /mnt/bricks/export/brick/testlog.txt 
[root@SN-0(matrix) /home/_rcpadmin]


Steps to Reproduce:
1. dd if=/dev/zero of=/mnt/export/testfull bs=10M count=1000
   ls /mnt/export/
2. tail -n 100 /var/log/syslog >/mnt/export/testlog.txt
   ls -al /mnt/export/
3. cat /mnt/export/testlog.txt 
4. ssh _rcpadmin@169.254.0.90
   ls -al /mnt/bricks/export/brick/
   cat /mnt/bricks/export/brick/testlog.txt 

Actual results:
In glusterfs file system the file can't be cat, it return No such file or directory, but file indeed is exist, glusterfs return a wrong error code 

Expected results:
the file can be success cat and don't return error

Additional info:

Error log of glusterFS
[2015-12-02 05:47:53.504915] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517071] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517120] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 1164110: WRITE => -1 (Input/output error)
[2015-12-02 05:47:53.517159] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517177] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.517200] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517215] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.517239] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517252] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517266] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517303] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 1164112: WRITE => -1 (No space left on device)
[2015-12-02 05:47:53.520701] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.524891] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.528614] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.532538] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.536883] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.540618] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.710057] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:55.433488] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 1164113: FLUSH() ERR => -1 (No space left on device)
[2015-12-02 05:55:06.742491] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:55:06.742693] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:55:08.466284] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 1172161: FLUSH() ERR => -1 (No space left on device)
[2015-12-02 06:02:00.935488] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-export-client-1: remote operation failed: No such file or directory. Path: /testlog.txt (8cd0d872-a383-4290-a330-e69d0b544862)
[2015-12-02 06:02:00.935523] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-export-client-0: remote operation failed: No such file or directory. Path: /testlog.txt (8cd0d872-a383-4290-a330-e69d0b544862)
[2015-12-02 06:02:00.935545] E [fuse-bridge.c:2095:fuse_open_resume] 0-glusterfs-fuse: 1180083: OPEN 8cd0d872-a383-4290-a330-e69d0b544862 resolution failed

Note You need to log in before you can comment on or make changes to this bug.