Bug 811514 - When disk is full, mkdir <dirname> succeeds but 'ls' doesn't list the directory entry
When disk is full, mkdir <dirname> succeeds but 'ls' doesn't list the directo...
Status: CLOSED WORKSFORME
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
pre-release
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: shishir gowda
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-11 06:08 EDT by M S Vishwanath Bhat
Modified: 2016-05-31 21:55 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-07 04:57:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description M S Vishwanath Bhat 2012-04-11 06:08:29 EDT
Description of problem:
When the disk is almost full, mkdir <dirname> succeeded but ls (or ls -a) doesn't list the directory entry. directory will not be listed even if some files are deleted to make some space. But I can cd into the dir and create some other files. But again rmdir of that directory fails.

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa33

How reproducible:
1/1

Steps to Reproduce:
1. Create and start a 2*2 distributed-replicated volume.
2. Now Create a large file(s) and make the disk full. (At least make one of the back-end disks full)
3. Now mkdir dir1
4. mkdir succeeds now do ls or ls -a
  
Actual results:
ls or ls -a do not list the directory. But I can cd into the directory and create some other files. But again rmdir of that directory fails.

Expected results:
ls or ls -a should list the directory. 

Additional info:

Entries from the fuse client log. 


[2012-04-11 05:02:51.212382] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:37:28.094445] W [client3_1-fops.c:330:client3_1_mkdir_cbk] 0-hosdu-client-0: remote operation failed: No space left on device. Path: /sumne
[2012-04-11 05:37:28.094509] W [client3_1-fops.c:330:client3_1_mkdir_cbk] 0-hosdu-client-1: remote operation failed: No space left on device. Path: /sumne
[2012-04-11 05:37:29.936199] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: /, reason: lookup detected pending operations
[2012-04-11 05:37:29.937080] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-hosdu-replicate-0: /: Performing conservative merge
[2012-04-11 05:37:29.942768] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on /
[2012-04-11 05:37:58.663575] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:37:58.665010] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:37:58.665878] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:38:03.717339] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:38:03.718905] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:38:03.719654] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:38:34.415924] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:38:34.417517] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:38:34.418158] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:38:42.491768] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:38:42.493212] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:38:42.494166] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:40:28.728897] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:40:28.728994] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:40:28.730550] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:47:12.658050] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:47:12.658808] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:47:12.658860] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:47:12.659532] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:06.371902] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:07.940029] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:07.940658] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:07.940731] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:07.941437] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:10.441842] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:12.062872] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:16.583794] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:16.583872] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:16.584550] W [fuse-bridge.c:1010:fuse_unlink_cbk] 0-glusterfs-fuse: 1522905318: RMDIR() /sumne => -1 (No such file or directory)
Comment 1 shishir gowda 2012-05-14 02:21:11 EDT
Is this bug still reproducible?
Comment 2 shishir gowda 2012-06-07 04:57:16 EDT
I am not able to reproduce the bug on release-3.3.

root@shishirng:/opt/qa# gluster volume info
 
Volume Name: new
Type: Distributed-Replicate
Volume ID: 9cd6dc71-9e78-4222-b933-aeace6f8ad26
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: sng:/tmp/exp1
Brick2: sng:/tmp/exp2
Brick3: sng:/tmp/exp3
Brick4: sng:/tmp/exp4


/dev/sda3       118G   69M  118G   1% /export
/dev/loop0       26M  1.6M   24M   7% /tmp/exp1
/dev/loop1       26M  1.6M   24M   7% /tmp/exp2
/dev/loop2       26M   26M   20K 100% /tmp/exp3 <--full
/dev/loop3       26M   26M   20K 100% /tmp/exp4 <--full

root@shishirng:/mnt/rep# mkdir dir
root@shishirng:/mnt/rep# ls -l
total 24300
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir
-rw-r--r-- 1 root root 25399296 Jun  7 14:22 testfile

root@shishirng:/mnt/rep# mkdir dir1
root@shishirng:/mnt/rep# ls -l
total 24300
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir
drwxr-xr-x 2 root root        6 Jun  7 14:25 dir1
-rw-r--r-- 1 root root 25399296 Jun  7 14:22 testfile

root@shishirng:/mnt/rep# mkdir dir2
root@shishirng:/mnt/rep# ls -l
total 24300
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir
drwxr-xr-x 2 root root        6 Jun  7 14:25 dir1
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir2
-rw-r--r-- 1 root root 25399296 Jun  7 14:22 testfile

root@shishirng:/mnt/rep# cd dir
root@shishirng:/mnt/rep/dir# ls


root@shishirng:/mnt/rep/dir# cd ..
root@shishirng:/mnt/rep# rm -rf *
root@shishirng:/mnt/rep# ls
root@shishirng:/mnt/rep#

Note You need to log in before you can comment on or make changes to this bug.