Description of problem: When the disk is almost full, mkdir <dirname> succeeded but ls (or ls -a) doesn't list the directory entry. directory will not be listed even if some files are deleted to make some space. But I can cd into the dir and create some other files. But again rmdir of that directory fails. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa33 How reproducible: 1/1 Steps to Reproduce: 1. Create and start a 2*2 distributed-replicated volume. 2. Now Create a large file(s) and make the disk full. (At least make one of the back-end disks full) 3. Now mkdir dir1 4. mkdir succeeds now do ls or ls -a Actual results: ls or ls -a do not list the directory. But I can cd into the directory and create some other files. But again rmdir of that directory fails. Expected results: ls or ls -a should list the directory. Additional info: Entries from the fuse client log. [2012-04-11 05:02:51.212382] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background data self-heal completed on /input/Story_of_Bert_wilson [2012-04-11 05:37:28.094445] W [client3_1-fops.c:330:client3_1_mkdir_cbk] 0-hosdu-client-0: remote operation failed: No space left on device. Path: /sumne [2012-04-11 05:37:28.094509] W [client3_1-fops.c:330:client3_1_mkdir_cbk] 0-hosdu-client-1: remote operation failed: No space left on device. Path: /sumne [2012-04-11 05:37:29.936199] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background entry self-heal triggered. path: /, reason: lookup detected pending operations [2012-04-11 05:37:29.937080] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-hosdu-replicate-0: /: Performing conservative merge [2012-04-11 05:37:29.942768] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background entry self-heal completed on / [2012-04-11 05:37:58.663575] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations [2012-04-11 05:37:58.665010] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson [2012-04-11 05:37:58.665878] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background data self-heal completed on /input/Story_of_Bert_wilson [2012-04-11 05:38:03.717339] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations [2012-04-11 05:38:03.718905] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson [2012-04-11 05:38:03.719654] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background data self-heal completed on /input/Story_of_Bert_wilson [2012-04-11 05:38:34.415924] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations [2012-04-11 05:38:34.417517] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson [2012-04-11 05:38:34.418158] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background data self-heal completed on /input/Story_of_Bert_wilson [2012-04-11 05:38:42.491768] I [afr-common.c:1329:afr_launch_self_heal] 0-hosdu-replicate-0: background data self-heal triggered. path: /input/Story_of_Bert_wilson, reason: lookup detected pending operations [2012-04-11 05:38:42.493212] I [afr-self-heal-data.c:776:afr_sh_data_fix] 0-hosdu-replicate-0: no active sinks for performing self-heal on file /input/Story_of_Bert_wilson [2012-04-11 05:38:42.494166] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background data self-heal completed on /input/Story_of_Bert_wilson [2012-04-11 05:40:28.728897] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:40:28.728994] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:40:28.730550] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:47:12.658050] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:47:12.658808] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:47:12.658860] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:47:12.659532] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:59:06.371902] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:59:07.940029] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:59:07.940658] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:59:07.940731] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:59:07.941437] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:59:10.441842] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:59:12.062872] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory [2012-04-11 05:59:16.583794] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:59:16.583872] W [client3_1-fops.c:2525:client3_1_opendir_cbk] 0-hosdu-client-1: remote operation failed: No such file or directory. Path: /sumne [2012-04-11 05:59:16.584550] W [fuse-bridge.c:1010:fuse_unlink_cbk] 0-glusterfs-fuse: 1522905318: RMDIR() /sumne => -1 (No such file or directory)
Is this bug still reproducible?
I am not able to reproduce the bug on release-3.3. root@shishirng:/opt/qa# gluster volume info Volume Name: new Type: Distributed-Replicate Volume ID: 9cd6dc71-9e78-4222-b933-aeace6f8ad26 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: sng:/tmp/exp1 Brick2: sng:/tmp/exp2 Brick3: sng:/tmp/exp3 Brick4: sng:/tmp/exp4 /dev/sda3 118G 69M 118G 1% /export /dev/loop0 26M 1.6M 24M 7% /tmp/exp1 /dev/loop1 26M 1.6M 24M 7% /tmp/exp2 /dev/loop2 26M 26M 20K 100% /tmp/exp3 <--full /dev/loop3 26M 26M 20K 100% /tmp/exp4 <--full root@shishirng:/mnt/rep# mkdir dir root@shishirng:/mnt/rep# ls -l total 24300 drwxr-xr-x 2 root root 12 Jun 7 14:25 dir -rw-r--r-- 1 root root 25399296 Jun 7 14:22 testfile root@shishirng:/mnt/rep# mkdir dir1 root@shishirng:/mnt/rep# ls -l total 24300 drwxr-xr-x 2 root root 12 Jun 7 14:25 dir drwxr-xr-x 2 root root 6 Jun 7 14:25 dir1 -rw-r--r-- 1 root root 25399296 Jun 7 14:22 testfile root@shishirng:/mnt/rep# mkdir dir2 root@shishirng:/mnt/rep# ls -l total 24300 drwxr-xr-x 2 root root 12 Jun 7 14:25 dir drwxr-xr-x 2 root root 6 Jun 7 14:25 dir1 drwxr-xr-x 2 root root 12 Jun 7 14:25 dir2 -rw-r--r-- 1 root root 25399296 Jun 7 14:22 testfile root@shishirng:/mnt/rep# cd dir root@shishirng:/mnt/rep/dir# ls root@shishirng:/mnt/rep/dir# cd .. root@shishirng:/mnt/rep# rm -rf * root@shishirng:/mnt/rep# ls root@shishirng:/mnt/rep#