Description of problem: Volume Name: test1 Type: Distributed-Replicate Volume ID: ae4f5ddc-dcf1-4298-b705-c71e81a5b12f Status: Stopped Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: 10.16.157.81:/home/test1-dr Brick2: 10.16.157.75:/home/test1-dr2 Brick3: 10.16.157.78:/home/test1-d2r Brick4: 10.16.157.21:/home/test1-d2r2 Brick5: 10.16.157.81:/home/test1-d3r Brick6: 10.16.157.75:/home/test1-d3r2 Brick7: 10.16.157.78:/home/test1-d4r Brick8: 10.16.157.21:/home/test1-d4r2 Brick9: 10.16.157.81:/home/test1-d5r Brick10: 10.16.157.75:/home/test1-d5r2 Brick11: 10.16.157.78:/home/test1-d6r Brick12: 10.16.157.21:/home/test1-d6r2 Options Reconfigured: features.quota: off geo-replication.indexing: on Below mentioned messages are found in /var/log/glusterfs/mnt-gluster-object-AUTH_test1.log file quite regularly, [2012-07-30 10:32:26.266966] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-0: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924) [2012-07-30 10:32:26.267075] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-1: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924) [2012-07-30 10:32:26.267123] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-8: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924) [2012-07-30 10:32:26.267253] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 2-test1-client-9: remote operation failed: Stale NFS file handle. Path: /cont78-2966/dir1/2966 (a15a528e-2b8e-4b36-844c-4c3d8b7e7924) Version-Release number of selected component (if applicable): [root@gqac028 glusterfs]# glusterfs -V glusterfs 3.3.0 built on Jul 19 2012 14:08:45 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. How reproducible: Happening regularly on one node Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: The REST API requests are coming to all the nodes of the cluster, though one nodes is only complaining about other logs.
Saurabh, this issue can happen in the case where file is changed on server after it was being accessed from one node (changed as in, re-created). as long as the application doesn't see any issue, this should be ok.
Saurabh, can you please answer comment#1? was application seeing any issues?
It's been a long time that I files this bug... so I can't presently remember the whole scenario.
[client3_1-fops.c:2630:client3_1_lookup_cbk] 0-Staging-client-3: remote operation failed: Stale NFS file handle. Path: /Q0/kacica/divoka (d376885d-3759-48a5-8d24-7f5a6842d940) I'm getting perl exception where we have: if ( -r $file ) on path divoka/oink/upload.zip So: I upload file and after that I'm doing stat call on it on another node. I get Stale NFS file handle error in log and stat fails even though after while I got to file and checked it's creation time, which was ~100ms before I got failed stat.
Saurabh, can you mention the steps to reproduce it.
This bug is one year old and presently I am not remembering the sequence of steps that led to display the error messages. Also this was found along with older UFO solution.