Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 844584

Summary:	logging: Stale NFS messages
Product:	[Community] GlusterFS	Reporter:	Saurabh <saujain>
Component:	logging	Assignee:	Susant Kumar Palai <spalai>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	pre-release	CC:	amarts, gluster-bugs, mzywusko, rfortier, saujain, sgowda, social
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-18 09:57:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Saurabh 2012-07-31 06:29:16 UTC

Description of problem:
Volume Name: test1
Type: Distributed-Replicate
Volume ID: ae4f5ddc-dcf1-4298-b705-c71e81a5b12f
Status: Stopped
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.16.157.81:/home/test1-dr
Brick2: 10.16.157.75:/home/test1-dr2
Brick3: 10.16.157.78:/home/test1-d2r
Brick4: 10.16.157.21:/home/test1-d2r2
Brick5: 10.16.157.81:/home/test1-d3r
Brick6: 10.16.157.75:/home/test1-d3r2
Brick7: 10.16.157.78:/home/test1-d4r
Brick8: 10.16.157.21:/home/test1-d4r2
Brick9: 10.16.157.81:/home/test1-d5r
Brick10: 10.16.157.75:/home/test1-d5r2
Brick11: 10.16.157.78:/home/test1-d6r
Brick12: 10.16.157.21:/home/test1-d6r2
Options Reconfigured:
features.quota: off
geo-replication.indexing: on

Below mentioned messages are found in /var/log/glusterfs/mnt-gluster-object-AUTH_test1.log file quite regularly,

[2012-07-30 10:32:26.266966] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-0: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924)
[2012-07-30 10:32:26.267075] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-1: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924)
[2012-07-30 10:32:26.267123] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-8: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924)
[2012-07-30 10:32:26.267253] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-9: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924)


Version-Release number of selected component (if applicable):

[root@gqac028 glusterfs]# glusterfs -V
glusterfs 3.3.0 built on Jul 19 2012 14:08:45
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

How reproducible:
Happening regularly on one node

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

The REST API requests are coming to all the nodes of the cluster, though one nodes is only complaining about other logs.

Comment 1 Amar Tumballi 2012-09-20 07:01:40 UTC

Saurabh, this issue can happen in the case where file is changed on server after it was being accessed from one node (changed as in, re-created). as long as the application doesn't see any issue, this should be ok.

Comment 2 Amar Tumballi 2012-11-29 11:21:08 UTC

Saurabh, can you please answer comment#1? was application seeing any issues?

Comment 3 Saurabh 2012-11-29 12:34:28 UTC

It's been a long time that I files this bug... so I can't presently remember the whole scenario.

Comment 4 Lukas Bezdicka 2013-06-05 23:08:19 UTC

[client3_1-fops.c:2630:client3_1_lookup_cbk] 0-Staging-client-3: remote operation failed: Stale NFS file handle. Path: /Q0/kacica/divoka (d376885d-3759-48a5-8d24-7f5a6842d940)

I'm getting perl exception where we have:
if ( -r $file )
on path divoka/oink/upload.zip

So:
I upload file and after that I'm doing stat call on it on another node. I get Stale NFS file handle error in log and stat fails even though after while I got to file and checked it's creation time, which was ~100ms before I got failed stat.

Comment 5 Susant Kumar Palai 2013-06-11 06:19:50 UTC

Saurabh, can you mention the steps to reproduce it.

Comment 6 Saurabh 2013-07-01 08:44:14 UTC

This bug is one year  old and presently I am not remembering the sequence of steps that led to display the error messages.

Also this was found along with older UFO solution.