958660 – afr: Fuse Client logs filled with afr_open_fd logs

Bug 958660 - afr: Fuse Client logs filled with afr_open_fd logs

Summary: afr: Fuse Client logs filled with afr_open_fd logs

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	glusterfs
Sub Component:
Version:	2.1
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	vsomyaju
QA Contact:	Rahul Hinduja
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-05-02 07:30 UTC by Rahul Hinduja
Modified:	2015-03-05 00:06 UTC (History)
CC List:	5 users (show)
Fixed In Version:	glusterfs-3.4.0.6rhs-1
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-09-23 22:35:25 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Rahul Hinduja 2013-05-02 07:30:12 UTC

Description of problem:
=======================

Fuse client log is completely filled with the following logs:

[2013-05-02 11:20:20.820615] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.822218] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.822880] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.823867] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.825094] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.826095] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.827210] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.828466] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.829586] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.830266] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.831934] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.832640] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.833503] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.834604] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.835874] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.837046] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.838019] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.839207] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.839919] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.841525] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.842237] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.843208] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.844223] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.845317] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.846631] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.847703] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.848766] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.849565] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.851154] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.851863] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.852845] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.853990] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.854979] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.856122] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.857351] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.858466] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.859226] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.860848] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.861558] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.862646] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.863654] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.864740] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.865957] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.866996] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.868129] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.869344] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.869415] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:20.871603] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:20.872728] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.873819] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.874921] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:20.876099] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc


Version-Release number of selected component (if applicable):
=============================================================

[root@darrel f]# rpm -qa | grep gluster
glusterfs-debuginfo-3.4.0.1rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.1rhs-1.el6rhs.x86_64
glusterfs-3.4.0.1rhs-1.el6rhs.x86_64
[root@darrel f]# 



How reproducible:
=================
1/1


Steps to Reproduce:
1. Create 6*2 volume 
2. Fuse mount on client
3. Start IO from the mount point.
4. While IO creation is in progress, unmount the brick directory from one server.
5. Try to create some file/directory from the fuse client 
  
Actual results:
===============
[2013-05-02 11:20:23.458980] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:23.460569] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:23.461364] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.462342] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.463380] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.464439] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.465592] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.466744] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.467851] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.468610] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e)
[2013-05-02 11:20:23.470061] I [afr-inode-write.c:526:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Opening fd 0x21d02bc
[2013-05-02 11:20:23.470831] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.471858] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.473019] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.474144] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.475157] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.476337] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.477466] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.478606] I [afr-inode-write.c:504:afr_open_fd_fix] 0-vol-dis-rep-replicate-0: Pause fd 0x21d02bc
[2013-05-02 11:20:23.478741] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-vol-dis-rep-client-0: remote operation failed: No such file or directory. Path: /f/test_file3 (60b6897d-b02b-4c27-baff-a02b49ea285e

Comment 4 Rahul Hinduja 2013-05-20 10:01:34 UTC

Verified with build:

[root@darrel ~]# rpm -qa | grep gluster 
glusterfs-rdma-3.4.0.8rhs-1.el6.x86_64
glusterfs-debuginfo-3.4.0.8rhs-1.el6.x86_64
glusterfs-devel-3.4.0.8rhs-1.el6.x86_64
glusterfs-fuse-3.4.0.8rhs-1.el6.x86_64
glusterfs-3.4.0.8rhs-1.el6.x86_64
[root@darrel ~]# 

[root@darrel ~]# cat /var/log/glusterfs/mnt-test-fuse.log | grep "afr_open_fd"
[root@darrel ~]#

Comment 5 Scott Haines 2013-09-23 22:35:25 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.