Bug 811514

Summary:	When disk is full, mkdir <dirname> succeeds but 'ls' doesn't list the directory entry
Product:	[Community] GlusterFS	Reporter:	M S Vishwanath Bhat <vbhat>
Component:	distribute	Assignee:	shishir gowda <sgowda>
Status:	CLOSED WORKSFORME	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	pre-release	CC:	gluster-bugs, mzywusko, nsathyan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-06-07 08:57:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description M S Vishwanath Bhat 2012-04-11 10:08:29 UTC

Description of problem:
When the disk is almost full, mkdir <dirname> succeeded but ls (or ls -a) doesn't list the directory entry. directory will not be listed even if some files are deleted to make some space. But I can cd into the dir and create some other files. But again rmdir of that directory fails.

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa33

How reproducible:
1/1

Steps to Reproduce:
1. Create and start a 2*2 distributed-replicated volume.
2. Now Create a large file(s) and make the disk full. (At least make one of the back-end disks full)
3. Now mkdir dir1
4. mkdir succeeds now do ls or ls -a
  
Actual results:
ls or ls -a do not list the directory. But I can cd into the directory and create some other files. But again rmdir of that directory fails.

Expected results:
ls or ls -a should list the directory. 

Additional info:

Entries from the fuse client log. 


[2012-04-11 05:02:51.212382] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:37:28.094445] W [client3_1-fops.c:330:client3_1_mkdir_cbk] 0-hosdu-client-0: remote operation failed: No space left on device. Path: /sumne
[2012-04-11 05:37:28.094509] W [client3_1-fops.c:330:client3_1_mkdir_cbk] 0-hosdu-client-1: remote operation failed: No space left on device. Path: /sumne
[2012-04-11 05:37:29.936199] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: /, reason: lookup detected pending operations
[2012-04-11 05:37:29.937080] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-hosdu-replicate-0: /: Performing conservative merge
[2012-04-11 05:37:29.942768] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on /
[2012-04-11 05:37:58.663575] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:37:58.665010] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:37:58.665878] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:38:03.717339] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:38:03.718905] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:38:03.719654] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:38:34.415924] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:38:34.417517] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:38:34.418158] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:38:42.491768] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations
[2012-04-11 05:38:42.493212] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson
[2012-04-11 05:38:42.494166] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on /input/Story_of_Bert_wilson
[2012-04-11 05:40:28.728897] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:40:28.728994] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:40:28.730550] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:47:12.658050] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:47:12.658808] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:47:12.658860] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:47:12.659532] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:06.371902] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:07.940029] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:07.940658] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:07.940731] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:07.941437] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:10.441842] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:12.062872] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
[2012-04-11 05:59:16.583794] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:16.583872] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne
[2012-04-11 05:59:16.584550] W [fuse-bridge.c:1010:fuse_unlink_cbk] 0-glusterfs-fuse: 1522905318: RMDIR() /sumne => -1 (No such file or directory)

Comment 1 shishir gowda 2012-05-14 06:21:11 UTC

Is this bug still reproducible?

Comment 2 shishir gowda 2012-06-07 08:57:16 UTC

I am not able to reproduce the bug on release-3.3.

root@shishirng:/opt/qa# gluster volume info
 
Volume Name: new
Type: Distributed-Replicate
Volume ID: 9cd6dc71-9e78-4222-b933-aeace6f8ad26
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: sng:/tmp/exp1
Brick2: sng:/tmp/exp2
Brick3: sng:/tmp/exp3
Brick4: sng:/tmp/exp4


/dev/sda3       118G   69M  118G   1% /export
/dev/loop0       26M  1.6M   24M   7% /tmp/exp1
/dev/loop1       26M  1.6M   24M   7% /tmp/exp2
/dev/loop2       26M   26M   20K 100% /tmp/exp3 <--full
/dev/loop3       26M   26M   20K 100% /tmp/exp4 <--full

root@shishirng:/mnt/rep# mkdir dir
root@shishirng:/mnt/rep# ls -l
total 24300
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir
-rw-r--r-- 1 root root 25399296 Jun  7 14:22 testfile

root@shishirng:/mnt/rep# mkdir dir1
root@shishirng:/mnt/rep# ls -l
total 24300
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir
drwxr-xr-x 2 root root        6 Jun  7 14:25 dir1
-rw-r--r-- 1 root root 25399296 Jun  7 14:22 testfile

root@shishirng:/mnt/rep# mkdir dir2
root@shishirng:/mnt/rep# ls -l
total 24300
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir
drwxr-xr-x 2 root root        6 Jun  7 14:25 dir1
drwxr-xr-x 2 root root       12 Jun  7 14:25 dir2
-rw-r--r-- 1 root root 25399296 Jun  7 14:22 testfile

root@shishirng:/mnt/rep# cd dir
root@shishirng:/mnt/rep/dir# ls


root@shishirng:/mnt/rep/dir# cd ..
root@shishirng:/mnt/rep# rm -rf *
root@shishirng:/mnt/rep# ls
root@shishirng:/mnt/rep#