Bug 974475 - afr: rm -rf of files which are in split brain is successful from fuse client but gives input/output error from NFS
afr: rm -rf of files which are in split brain is successful from fuse client ...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Pranith Kumar K
Rahul Hinduja
:
Depends On:
Blocks: 974972
  Show dependency treegraph
 
Reported: 2013-06-14 05:00 EDT by Rahul Hinduja
Modified: 2013-09-23 18:35 EDT (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.12rhs.beta6-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 974972 (view as bug list)
Environment:
Last Closed: 2013-09-23 18:35:38 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul Hinduja 2013-06-14 05:00:04 EDT
Description of problem:
=======================

When tried to delete the files/directories that are in split brain from Fuse and NFS mount, it is successful from fuse but returns input/output error from NFS.

From Fuse Client:
=================

[f]# rm -rf * 
[f]# 
[f]# ls
[f]# 
[f]# pwd
/mnt/vol-dis-rep/f
[f]# 

From NFS Client:
================

[n]# ls
dir.1  dir.10  dir.2  dir.3  dir.4  dir.5  dir.6  dir.7  dir.8  dir.9  file.1  file.10  file.2  file.3  file.4  file.5  file.6  file.7  file.8  file.9
[n]# 
[n]# rm -rf * 
rm: cannot remove `dir.1/file.1': Input/output error
rm: cannot remove `dir.1/file.2': Input/output error
rm: cannot remove `dir.1/file.3': Input/output error
rm: cannot remove `dir.1/file.4': Input/output error
rm: cannot remove `dir.1/file.5': Input/output error
rm: cannot remove `dir.10/file.2': Input/output error
rm: cannot remove `dir.10/file.3': Input/output error
rm: cannot remove `dir.10/file.4': Input/output error
rm: cannot remove `dir.10/file.5': Input/output error
rm: cannot remove `dir.10/file.1': Input/output error
rm: cannot remove `dir.2/file.2': Input/output error
rm: cannot remove `dir.2/file.3': Input/output error
rm: cannot remove `dir.2/file.4': Input/output error
rm: cannot remove `dir.2/file.5': Input/output error
rm: cannot remove `dir.2/file.1': Input/output error
rm: cannot remove `dir.3/file.2': Input/output error
rm: cannot remove `dir.3/file.3': Input/output error
rm: cannot remove `dir.3/file.4': Input/output error
rm: cannot remove `dir.3/file.5': Input/output error
rm: cannot remove `dir.3/file.1': Input/output error
rm: cannot remove `dir.4/file.1': Input/output error
rm: cannot remove `dir.4/file.2': Input/output error
rm: cannot remove `dir.4/file.3': Input/output error
rm: cannot remove `dir.4/file.4': Input/output error
rm: cannot remove `dir.4/file.5': Input/output error
rm: cannot remove `dir.5/file.2': Input/output error
rm: cannot remove `dir.5/file.3': Input/output error
rm: cannot remove `dir.5/file.4': Input/output error
rm: cannot remove `dir.5/file.5': Input/output error
rm: cannot remove `dir.5/file.1': Input/output error
rm: cannot remove `dir.6/file.1': Input/output error
rm: cannot remove `dir.6/file.2': Input/output error
rm: cannot remove `dir.6/file.3': Input/output error
rm: cannot remove `dir.6/file.4': Input/output error
rm: cannot remove `dir.6/file.5': Input/output error
rm: cannot remove `dir.7/file.1': Input/output error
rm: cannot remove `dir.7/file.2': Input/output error
rm: cannot remove `dir.7/file.3': Input/output error
rm: cannot remove `dir.7/file.4': Input/output error
rm: cannot remove `dir.7/file.5': Input/output error
rm: cannot remove `dir.8/file.2': Input/output error
rm: cannot remove `dir.8/file.3': Input/output error
rm: cannot remove `dir.8/file.4': Input/output error
rm: cannot remove `dir.8/file.5': Input/output error
rm: cannot remove `dir.8/file.1': Input/output error
rm: cannot remove `dir.9/file.1': Input/output error
rm: cannot remove `dir.9/file.2': Input/output error
rm: cannot remove `dir.9/file.3': Input/output error
rm: cannot remove `dir.9/file.4': Input/output error
rm: cannot remove `dir.9/file.5': Input/output error
rm: cannot remove `file.1': Input/output error
rm: cannot remove `file.10': Input/output error
rm: cannot remove `file.2': Input/output error
rm: cannot remove `file.3': Input/output error
rm: cannot remove `file.4': Input/output error
rm: cannot remove `file.5': Input/output error
rm: cannot remove `file.6': Input/output error
rm: cannot remove `file.7': Input/output error
rm: cannot remove `file.8': Input/output error
rm: cannot remove `file.9': Input/output error

[n]# pwd
/mnt/nvol/n
[n]# 

Version-Release number of selected component (if applicable):
=============================================================

[n]# rpm -qa | grep gluster
glusterfs-fuse-3.4.0.9rhs-1.el6.x86_64
glusterfs-3.4.0.9rhs-1.el6.x86_64
glusterfs-debuginfo-3.4.0.9rhs-1.el6.x86_64
glusterfs-devel-3.4.0.9rhs-1.el6.x86_64
glusterfs-rdma-3.4.0.9rhs-1.el6.x86_64
[n]# 


How reproducible:
=================
1/1


Steps Carried:
==============

1. Created 6*2 volume from 4 servers
2. Mounted on Client (NFS and FUSE)
3. Created directory names f and n from fuse mount 
4. From fuse mount cd to f
5. From NFS mount cd to n
6. Metadata,data and entry self heal were set to off
"for i in {'metadata','data','entry'} ; do gluster volume set <volume_name> $i-self-heal off ; done"
7. From both the mounted directories executed the following command to create directories, files within directories and files:

for i in `seq 1 10` ; do mkdir dir.$i ; for j in `seq 1 5` ; do dd if=/dev/urandom of=dir.$i/file.$j bs=1K count=1 ; done ; dd if=/dev/urandom of=file.$i bs=1k count=1 ; done

8. Set the self-heal-daemon to off

gluster volume set <vol_name> self-heal-daemon off

9. Brought down all bricks from server 2 and server 4 (kill -9)
10. Modified the content of files from both the mounted directories (f and n)

for i in `seq 1 10` ; do for j in `seq 1 5` ; do dd if=/dev/urandom of=dir.$i/file.$j bs=1M count=1 ; done ; dd if=/dev/urandom of=file.$i bs=1k count=1 ; done

10. Brought back the bricks of server 2 and server 4 using "gluster volume start <vol-name> force"

11. Killed the brick processes of server 1 and server 3 which were earlier UP.

12. Modify the content of files from fuse and nfs mounted directories (f and n)

for i in `seq 1 10` ; do for j in `seq 1 5` ; do dd if=/dev/urandom of=dir.$i/file.$j bs=1M count=1 ; done ; dd if=/dev/urandom of=file.$i bs=1k count=1 ; done

13. Modification of files should be successful.

14. Remove the files from Fuse and NFS mounted directory. NFS complains input/output error but removal from fuse is successful.


Actual results:
===============

From Fuse Client:
=================

[f]# rm -rf * 
[f]# 
[f]# ls
[f]# 
[f]# pwd
/mnt/vol-dis-rep/f
[f]# 

From NFS Client:
================

[n]# ls
dir.1  dir.10  dir.2  dir.3  dir.4  dir.5  dir.6  dir.7  dir.8  dir.9  file.1  file.10  file.2  file.3  file.4  file.5  file.6  file.7  file.8  file.9
[n]# 
[n]# rm -rf * 
rm: cannot remove `dir.1/file.1': Input/output error
rm: cannot remove `dir.1/file.2': Input/output error


Expected results:
=================

Should be consistent between Fuse and NFS protocols
Comment 5 Scott Haines 2013-09-23 18:35:38 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.