Bug 958660

Summary:	afr: Fuse Client logs filled with afr_open_fd logs
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	glusterfs	Assignee:	vsomyaju
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	high	Docs Contact:
Priority:	medium
Version:	2.1	CC:	amarts, nsathyan, pkarampu, rhs-bugs, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0.6rhs-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-09-23 22:35:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rahul Hinduja 2013-05-02 07:30:12 UTC

Description of problem:
=======================

Fuse client log is completely filled with the following logs:

[2013-05-02 11:20:20.820615] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.822218] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.822880] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.823867] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.825094] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.826095] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.827210] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.828466] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.829586] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.830266] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.831934] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.832640] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.833503] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.834604] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.835874] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.837046] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.838019] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.839207] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.839919] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.841525] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.842237] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.843208] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.844223] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.845317] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.846631] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.847703] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.848766] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.849565] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.851154] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.851863] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.852845] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.853990] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.854979] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.856122] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.857351] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.858466] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.859226] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.860848] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.861558] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.862646] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.863654] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.864740] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.865957] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.866996] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.868129] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.869344] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.869415] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.871603] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.872728] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.873819] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.874921] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.876099] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc


Version-Release number of selected component (if applicable):
=============================================================

[root@darrel f]# rpm -qa | grep gluster
glusterfs-debuginfo-3.4.0.1rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.1rhs-1.el6rhs.x86_64
glusterfs-3.4.0.1rhs-1.el6rhs.x86_64
[root@darrel f]# 



How reproducible:
=================
1/1


Steps to Reproduce:
1. Create 6*2 volume 
2. Fuse mount on client
3. Start IO from the mount point.
4. While IO creation is in progress, unmount the brick directory from one server.
5. Try to create some file/directory from the fuse client 
  
Actual results:
===============
[2013-05-02 11:20:23.458980] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:23.460569] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:23.461364] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.462342] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.463380] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.464439] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.465592] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.466744] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.467851] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.468610] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:23.470061] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:23.470831] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.471858] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.473019] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.474144] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.475157] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.476337] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.477466] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.478606] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.478741] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e

Comment 4 Rahul Hinduja 2013-05-20 10:01:34 UTC

Verified with build:

[root@darrel ~]# rpm -qa | grep gluster 
glusterfs-rdma-3.4.0.8rhs-1.el6.x86_64
glusterfs-debuginfo-3.4.0.8rhs-1.el6.x86_64
glusterfs-devel-3.4.0.8rhs-1.el6.x86_64
glusterfs-fuse-3.4.0.8rhs-1.el6.x86_64
glusterfs-3.4.0.8rhs-1.el6.x86_64
[root@darrel ~]# 

[root@darrel ~]# cat /var/log/glusterfs/mnt-test-fuse.log | grep "afr_open_fd"
[root@darrel ~]#

Comment 5 Scott Haines 2013-09-23 22:35:25 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html