Bug 809083

Summary:	Iozone failed with stale NFS file handle while running iozone on multiple clients on a geo-replication setup.
Product:	[Community] GlusterFS	Reporter:	Vijaykumar Koppad <vkoppad>
Component:	core	Assignee:	Amar Tumballi <amarts>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	mainline	CC:	bbandari, gluster-bugs, vraman
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	810076 (view as bug list)		Environment:
Last Closed:	2012-11-22 07:13:07 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Vijaykumar Koppad 2012-04-02 12:50:29 UTC

Description of problem:
In geo-replication setup with three slaves, while running running iozone on 5 clients simultaneously, one of the iozone failed. 

the client log said . 
[2012-03-30 00:57:00.049633] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
[2012-03-30 00:57:00.049687] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
[2012-03-30 00:57:04.036226] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
[2012-03-30 00:57:04.036425] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
[2012-03-30 00:57:04.036865] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.54:24010, attached to remote volume '/root/bricks/m1'.
[2012-03-30 00:57:04.036900] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2012-03-30 00:57:04.037853] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
[2012-03-30 00:57:04.042650] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
[2012-03-30 00:57:04.042848] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
[2012-03-30 00:57:04.043226] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.54:24011, attached to remote volume '/root/bricks/m2'.
[2012-03-30 00:57:04.043259] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2012-03-30 00:57:04.051401] I [fuse-bridge.c:4081:fuse_graph_setup] 0-fuse: switched to graph 0
[2012-03-30 00:57:04.051600] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
[2012-03-30 00:57:04.051942] I [fuse-bridge.c:3358:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2012-04-01 23:47:00.675108] W [client3_1-fops.c:419:client3_1_open_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: /iozone.tmp
[2012-04-01 23:47:00.695656] W [client3_1-fops.c:1054:client3_1_getxattr_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
[2012-04-01 23:47:00.706529] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-master-client-1: remote operation failed: Invalid argument. Path: /iozone.tmp
[2012-04-01 23:47:00.706779] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-master-client-0: remote operation failed: Invalid argument. Path: /iozone.tmp
[2012-04-01 23:47:00.706815] I [dht-layout.c:600:dht_layout_normalize] 0-master-dht: found anomalies in /iozone.tmp. holes=1 overlaps=0
[2012-04-01 23:47:00.706867] W [fuse-bridge.c:694:fuse_fd_cbk] 0-glusterfs-fuse: 99: OPEN() /iozone.tmp => -1 (No such file or directory)
[2012-04-01 23:47:15.170440] W [client3_1-fops.c:2607:client3_1_lookup_cbk] 0-master-client-0: remote operation failed: Stale NFS file handle. Path: /iozone.tmp
 

How reproducible: not often 


Steps to Reproduce:
1.Create a geo-rep setup with more slaves .
2.Run iozone on many clients simultaneously(iozone -a -f <filename>)

Comment 1 Amar Tumballi 2012-04-13 07:24:33 UTC

iozone on many clients with same filename? that may not be a valid run...

Comment 2 Amar Tumballi 2012-04-17 11:04:37 UTC

need information from vijaykumar koppad regarding comment #1

Comment 3 Vijaykumar Koppad 2012-04-23 11:22:27 UTC

It was not on the same file . Iozone won't allow to run many iozone tests on single file . That is why i used the -f option in iozone like "iozone -a -f <filename> ". Do you think that  "remote operation failed: Stale NFS file handle" 
is not a bug?.

Comment 4 Amar Tumballi 2012-09-18 05:45:15 UTC

Koppad, please see if this issue still happens with 3.3.0 or higher versions. If not, would like to close the bug.

Comment 5 Vijaykumar Koppad 2012-11-22 07:13:07 UTC

I tested it on 3.3.0.5rhs-38.el6rhs.x86_64.  It is working fine .