Bug 1287440 - When GlusterFS mount volume is full, create a new file and read the file, the error code shows No such file or directory, but the file indeed is exist
When GlusterFS mount volume is full, create a new file and read the file, the...
Status: CLOSED EOL
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core (Show other bugs)
unspecified
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
Anoop
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-02 01:13 EST by coyang
Modified: 2017-11-07 04:36 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-07 04:36:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description coyang 2015-12-02 01:13:09 EST
Description of problem:


Version-Release number of selected component (if applicable):
# glusterfs -V
glusterfs 3.6.2 built on Sep  1 2015 13:55:09


How reproducible:
Pre-condition: 
on client node, /mnt/export is mount by glusterfs
# df |grep export
169.254.0.90:/export      4782720   30464   4473728   1% /mnt/export
[root@CFPU-0(matrix) /home/_rcpadmin]
# mount |grep export
169.254.0.90:/export on /mnt/export type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

1. use dd make large file in /mnt/export which result in /mnt/export full
[root@CFPU-0(matrix) /home/_rcpadmin]
# dd if=/dev/zero of=/mnt/export/testfull bs=10M count=1000
dd: error writing ‘/mnt/export/testfull’: No space left on device
dd: closing output file ‘/mnt/export/testfull’: No space left on device
[root@CFPU-0(matrix) /home/_rcpadmin]
# ls /mnt/export/
fsconfigure.pid  symptomreport	test2.log  testfull  test.log  trace
[root@CFPU-0(matrix) /home/_rcpadmin]
# df /mnt/export/
Filesystem           1K-blocks    Used Available Use% Mounted on
169.254.0.90:/export   4782720 4766336         0 100% /mnt/export

2. tail log to a new log named testlog.txt,
[root@CFPU-0(matrix) /home/_rcpadmin]
# tail -n 100 /var/log/syslog >/mnt/export/testlog.txt
tail: write error: No space left on device
[root@CFPU-0(matrix) /home/_rcpadmin]

3. cat the file return No such file or directory, but ls shows the file is exist

[root@CFPU-0(matrix) /home/_rcpadmin]
# ls -al /mnt/export/
total 4735934
drwxrws---  5 root _nokfsuifileshare       4096 Dec  2 13:55 .
drwxr-xr-x 17 root root                    4096 Nov 30 16:07 ..
-rwxr-xr-x  1 root _nokfsuifileshare         17 Dec  1 21:08 fsconfigure.pid
drwxr-x---  2 root _nokfsuitracelog        4096 Nov 30 16:03 symptomreport
-rw-r--r--  1 root _nokfsuifileshare       1861 Dec  1 20:43 test2.log
-rw-r--r--  1 root _nokfsuifileshare 4849561600 Dec  2 13:47 testfull
-rw-r--r--  1 root _nokfsuifileshare      14906 Dec  1 20:43 test.log
-rw-r--r--  0 root _nokfsuifileshare          0 Dec  2 13:55 testlog.txt
drwxr-s---  2 root _nokfsuitracelog        4096 Nov 30 16:03 trace
[root@CFPU-0(matrix) /home/_rcpadmin]
# cat /mnt/export/testlog.txt 
cat: /mnt/export/testlog.txt: No such file or directory

4. ssh to storage node, the file is exist on /mnt/bricks/export/, cat can be success execute on local ext4 file system
# ssh _rcpadmin@169.254.0.90
Password: 
[root@SN-0(matrix) /home/_rcpadmin]
# ls -al /mnt/bricks/export/
brick/      lost+found/ 
[root@SN-0(matrix) /home/_rcpadmin]        
[root@SN-0(matrix) /home/_rcpadmin]
# ls -al /mnt/bricks/export/brick/
total 4735968
drwxrws---   5 root _nokfsuifileshare       4096 Dec  2 13:55 .
drwxr-xr-x   4 root root                    4096 Nov 30 15:59 ..
-rwxr-xr-x   2 root _nokfsuifileshare         17 Dec  1 21:08 fsconfigure.pid
drw------- 261 root root                    4096 Dec  1 11:45 .glusterfs
drwxr-x---   2 root _nokfsuitracelog        4096 Nov 30 16:03 symptomreport
-rw-r--r--   2 root _nokfsuifileshare       1861 Dec  1 20:43 test2.log
-rw-r--r--   2 root _nokfsuifileshare 4849561600 Dec  2 13:47 testfull
-rw-r--r--   2 root _nokfsuifileshare      14906 Dec  1 20:43 test.log
-rw-r--r--   1 root _nokfsuifileshare          0 Dec  2 13:55 testlog.txt
drwxr-s---   2 root _nokfsuitracelog        4096 Nov 30 16:03 trace


# cat /mnt/bricks/export/brick/testlog.txt 
[root@SN-0(matrix) /home/_rcpadmin]


Steps to Reproduce:
1. dd if=/dev/zero of=/mnt/export/testfull bs=10M count=1000
   ls /mnt/export/
2. tail -n 100 /var/log/syslog >/mnt/export/testlog.txt
   ls -al /mnt/export/
3. cat /mnt/export/testlog.txt 
4. ssh _rcpadmin@169.254.0.90
   ls -al /mnt/bricks/export/brick/
   cat /mnt/bricks/export/brick/testlog.txt 

Actual results:
In glusterfs file system the file can't be cat, it return No such file or directory, but file indeed is exist, glusterfs return a wrong error code 

Expected results:
the file can be success cat and don't return error

Additional info:

Error log of glusterFS
[2015-12-02 05:47:53.504915] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517071] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517120] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 1164110: WRITE => -1 (Input/output error)
[2015-12-02 05:47:53.517159] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517177] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.517200] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517215] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.517239] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517252] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517266] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:53.517303] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 1164112: WRITE => -1 (No space left on device)
[2015-12-02 05:47:53.520701] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.524891] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.528614] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.532538] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.536883] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.540618] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:47:53.710057] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:47:55.433488] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 1164113: FLUSH() ERR => -1 (No space left on device)
[2015-12-02 05:55:06.742491] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-1: remote operation failed: No space left on device
[2015-12-02 05:55:06.742693] W [client-rpc-fops.c:851:client3_3_writev_cbk] 0-export-client-0: remote operation failed: No space left on device
[2015-12-02 05:55:08.466284] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 1172161: FLUSH() ERR => -1 (No space left on device)
[2015-12-02 06:02:00.935488] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-export-client-1: remote operation failed: No such file or directory. Path: /testlog.txt (8cd0d872-a383-4290-a330-e69d0b544862)
[2015-12-02 06:02:00.935523] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-export-client-0: remote operation failed: No such file or directory. Path: /testlog.txt (8cd0d872-a383-4290-a330-e69d0b544862)
[2015-12-02 06:02:00.935545] E [fuse-bridge.c:2095:fuse_open_resume] 0-glusterfs-fuse: 1180083: OPEN 8cd0d872-a383-4290-a330-e69d0b544862 resolution failed
Comment 2 coyang 2017-11-07 04:36:56 EST
close this pr, since it's very old release, in latest 3.10.6 release, this problem can't be reproduced.

Note You need to log in before you can comment on or make changes to this bug.