Description of problem:
=======================

When tried to delete the files/directories that are in split brain from Fuse and NFS mount, it is successful from fuse but returns input/output error from NFS.

From Fuse Client:
=================

[f]# rm -rf * 
[f]# 
[f]# ls
[f]# 
[f]# pwd
/mnt/vol-dis-rep/f
[f]# 

From NFS Client:
================

[n]# ls
dir.1  dir.10  dir.2  dir.3  dir.4  dir.5  dir.6  dir.7  dir.8  dir.9  file.1  file.10  file.2  file.3  file.4  file.5  file.6  file.7  file.8  file.9
[n]# 
[n]# rm -rf * 
rm: cannot remove `dir.1/file.1': Input/output error
rm: cannot remove `dir.1/file.2': Input/output error
rm: cannot remove `dir.1/file.3': Input/output error
rm: cannot remove `dir.1/file.4': Input/output error
rm: cannot remove `dir.1/file.5': Input/output error
rm: cannot remove `dir.10/file.2': Input/output error
rm: cannot remove `dir.10/file.3': Input/output error
rm: cannot remove `dir.10/file.4': Input/output error
rm: cannot remove `dir.10/file.5': Input/output error
rm: cannot remove `dir.10/file.1': Input/output error
rm: cannot remove `dir.2/file.2': Input/output error
rm: cannot remove `dir.2/file.3': Input/output error
rm: cannot remove `dir.2/file.4': Input/output error
rm: cannot remove `dir.2/file.5': Input/output error
rm: cannot remove `dir.2/file.1': Input/output error
rm: cannot remove `dir.3/file.2': Input/output error
rm: cannot remove `dir.3/file.3': Input/output error
rm: cannot remove `dir.3/file.4': Input/output error
rm: cannot remove `dir.3/file.5': Input/output error
rm: cannot remove `dir.3/file.1': Input/output error
rm: cannot remove `dir.4/file.1': Input/output error
rm: cannot remove `dir.4/file.2': Input/output error
rm: cannot remove `dir.4/file.3': Input/output error
rm: cannot remove `dir.4/file.4': Input/output error
rm: cannot remove `dir.4/file.5': Input/output error
rm: cannot remove `dir.5/file.2': Input/output error
rm: cannot remove `dir.5/file.3': Input/output error
rm: cannot remove `dir.5/file.4': Input/output error
rm: cannot remove `dir.5/file.5': Input/output error
rm: cannot remove `dir.5/file.1': Input/output error
rm: cannot remove `dir.6/file.1': Input/output error
rm: cannot remove `dir.6/file.2': Input/output error
rm: cannot remove `dir.6/file.3': Input/output error
rm: cannot remove `dir.6/file.4': Input/output error
rm: cannot remove `dir.6/file.5': Input/output error
rm: cannot remove `dir.7/file.1': Input/output error
rm: cannot remove `dir.7/file.2': Input/output error
rm: cannot remove `dir.7/file.3': Input/output error
rm: cannot remove `dir.7/file.4': Input/output error
rm: cannot remove `dir.7/file.5': Input/output error
rm: cannot remove `dir.8/file.2': Input/output error
rm: cannot remove `dir.8/file.3': Input/output error
rm: cannot remove `dir.8/file.4': Input/output error
rm: cannot remove `dir.8/file.5': Input/output error
rm: cannot remove `dir.8/file.1': Input/output error
rm: cannot remove `dir.9/file.1': Input/output error
rm: cannot remove `dir.9/file.2': Input/output error
rm: cannot remove `dir.9/file.3': Input/output error
rm: cannot remove `dir.9/file.4': Input/output error
rm: cannot remove `dir.9/file.5': Input/output error
rm: cannot remove `file.1': Input/output error
rm: cannot remove `file.10': Input/output error
rm: cannot remove `file.2': Input/output error
rm: cannot remove `file.3': Input/output error
rm: cannot remove `file.4': Input/output error
rm: cannot remove `file.5': Input/output error
rm: cannot remove `file.6': Input/output error
rm: cannot remove `file.7': Input/output error
rm: cannot remove `file.8': Input/output error
rm: cannot remove `file.9': Input/output error

[n]# pwd
/mnt/nvol/n
[n]# 

Version-Release number of selected component (if applicable):
=============================================================

[n]# rpm -qa | grep gluster
glusterfs-fuse-3.4.0.9rhs-1.el6.x86_64
glusterfs-3.4.0.9rhs-1.el6.x86_64
glusterfs-debuginfo-3.4.0.9rhs-1.el6.x86_64
glusterfs-devel-3.4.0.9rhs-1.el6.x86_64
glusterfs-rdma-3.4.0.9rhs-1.el6.x86_64
[n]# 


How reproducible:
=================
1/1


Steps Carried:
==============

1. Created 6*2 volume from 4 servers
2. Mounted on Client (NFS and FUSE)
3. Created directory names f and n from fuse mount 
4. From fuse mount cd to f
5. From NFS mount cd to n
6. Metadata,data and entry self heal were set to off
"for i in {'metadata','data','entry'} ; do gluster volume set <volume_name> $i-self-heal off ; done"
7. From both the mounted directories executed the following command to create directories, files within directories and files:

for i in `seq 1 10` ; do mkdir dir.$i ; for j in `seq 1 5` ; do dd if=/dev/urandom of=dir.$i/file.$j bs=1K count=1 ; done ; dd if=/dev/urandom of=file.$i bs=1k count=1 ; done

8. Set the self-heal-daemon to off

gluster volume set <vol_name> self-heal-daemon off

9. Brought down all bricks from server 2 and server 4 (kill -9)
10. Modified the content of files from both the mounted directories (f and n)

for i in `seq 1 10` ; do for j in `seq 1 5` ; do dd if=/dev/urandom of=dir.$i/file.$j bs=1M count=1 ; done ; dd if=/dev/urandom of=file.$i bs=1k count=1 ; done

10. Brought back the bricks of server 2 and server 4 using "gluster volume start <vol-name> force"

11. Killed the brick processes of server 1 and server 3 which were earlier UP.

12. Modify the content of files from fuse and nfs mounted directories (f and n)

for i in `seq 1 10` ; do for j in `seq 1 5` ; do dd if=/dev/urandom of=dir.$i/file.$j bs=1M count=1 ; done ; dd if=/dev/urandom of=file.$i bs=1k count=1 ; done

13. Modification of files should be successful.

14. Remove the files from Fuse and NFS mounted directory. NFS complains input/output error but removal from fuse is successful.


Actual results:
===============

From Fuse Client:
=================

[f]# rm -rf * 
[f]# 
[f]# ls
[f]# 
[f]# pwd
/mnt/vol-dis-rep/f
[f]# 

From NFS Client:
================

[n]# ls
dir.1  dir.10  dir.2  dir.3  dir.4  dir.5  dir.6  dir.7  dir.8  dir.9  file.1  file.10  file.2  file.3  file.4  file.5  file.6  file.7  file.8  file.9
[n]# 
[n]# rm -rf * 
rm: cannot remove `dir.1/file.1': Input/output error
rm: cannot remove `dir.1/file.2': Input/output error


Expected results:
=================

Should be consistent between Fuse and NFS protocols