Bug 996089

Summary: AFR: stat failed with `Transport endpoint is not connected' while accessed with 3.3 and 3.4 clients
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rahul Hinduja <rhinduja>
Component: glusterfsAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED ERRATA QA Contact: Rahul Hinduja <rhinduja>
Severity: high Docs Contact:
Priority: low    
Version: 2.1CC: aavati, amarts, kaushal, nsathyan, pkarampu, rabhat, rhs-bugs, sgowda, shaines, surs, vbellur
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.4.0.22rhs-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-23 22:29:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 996987    
Bug Blocks:    

Description Rahul Hinduja 2013-08-12 11:41:22 UTC
Description of problem:
=======================
Hit these error while mv the files from fuse mount of bigbend client, these files were originally written by anshi client.

mv: cannot stat `E_file.2': Transport endpoint is not connected
mv: cannot stat `E_file.4': Transport endpoint is not connected


Following are the details of setup and steps that hit this issue.

Setup:
======
I have 6*2 volume mounted on two clients with fuse and nfs mounts each (say client1 and client2), where client1 is RHEL6.4/3.4 and client2 is RHEL6.2/3.3 bits

Steps:
======
1. created directories a,b,c and d from the fuse mount of client2

2. cd to a from fuse mount of client1 and cd to b from fuse mount of client2

3. cd to c from nfs mount of client1 and cd to d from nfs mount of client2

4. run the script1.sh from all the directories a,b,c and d. script1.sh creates files and directories.

5. once the script1.sh is completed on both the clients, killall glusterd glusterfs glusterfsd on server2 and server3

6. now, cd to "b" from fuse mount of client1 and cd to "a" from fuse mount of client2 and run script2.sh which does the modification on the existing files and directories.

7. On fuse mount of client1 which was modifying the content of directory b on which the original data was written by client2 hit these error

[root@tia b]# ./script2.sh 
mv: cannot stat `E_file.2': Transport endpoint is not connected
mv: cannot stat `E_file.4': Transport endpoint is not connected
mv: cannot stat `E_file.7': Transport endpoint is not connected
mv: cannot stat `E_file.10': Transport endpoint is not connected
mv: cannot stat `E_file.16': Transport endpoint is not connected
mv: cannot stat `E_file.19': Transport endpoint is not connected
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0435713 s, 24.1 MB/s
2+0 records in
2+0 records out

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-fuse-3.4.0.18rhs-1.el6_4.x86_64
glusterfs-3.4.0.18rhs-1.el6_4.x86_64
glusterfs-libs-3.4.0.18rhs-1.el6_4.x86_64
glusterfs-debuginfo-3.4.0.18rhs-1.el6_4.x86_64

Actual results:
===============

[root@tia b]# ./script2.sh 
mv: cannot stat `E_file.2': Transport endpoint is not connected
mv: cannot stat `E_file.4': Transport endpoint is not connected
mv: cannot stat `E_file.7': Transport endpoint is not connected
mv: cannot stat `E_file.10': Transport endpoint is not connected
mv: cannot stat `E_file.16': Transport endpoint is not connected
mv: cannot stat `E_file.19': Transport endpoint is not connected
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0435713 s, 24.1 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.0210682 s, 99.5 MB/s
3+0 records in
3+0 records out
3145728 bytes (3.1 MB) copied, 0.0350536 s, 89.7 MB/s
4+0 records in
4+0 records out
4194304 bytes (4.2 MB) copied, 0.0414022 s, 101 MB/s
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.0572409 s, 91.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.03495 s, 30.0 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.0208142 s, 101 MB/s
3+0 records in
3+0 records out

Additional info:
================

There were 50 files with E_file.{1..50} and script2 was trying to mv the files from E_file.{1..20}. It hit the error on files E_files.{2,4,7,10,16,19} and rest is successful.

client log reported:
[2013-08-12 07:42:14.608791] W [fuse-bridge.c:1133:fuse_attr_cbk] 0-glusterfs-fuse: 799080: STAT() /b/E_file.2 => -1 (Transport endpoint is not connected)

nothing else is reported on any other logs.

Comment 6 Rahul Hinduja 2013-08-26 12:34:02 UTC
Verified with build: glusterfs-server-3.4.0.22rhs-2.el6rhs.x86_64 . The issue is not observed with the similar steps. Marking it verified.

Comment 7 Scott Haines 2013-09-23 22:29:51 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html