Bug 1228100 - Disperse volume: brick logs are getting filled with "anonymous fd creation failed" messages
Summary: Disperse volume: brick logs are getting filled with "anonymous fd creation fa...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: disperse
Version: 3.7.0
Hardware: All
OS: All
medium
high
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On: 1202758
Blocks: qe_tracker_everglades 1224136
TreeView+ depends on / blocked
 
Reported: 2015-06-04 08:32 UTC by Ashish Pandey
Modified: 2015-06-20 09:49 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.7.2
Doc Type: Bug Fix
Doc Text:
Clone Of: 1202758
Environment:
Last Closed: 2015-06-20 09:49:52 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Ashish Pandey 2015-06-04 08:32:02 UTC
+++ This bug was initially created as a clone of Bug #1202758 +++

Description of problem:
======================= 

Brick logs are getting filled with below messages and the filesystem is full within no time.

[2015-03-17 11:56:04.171990] W [server-resolve.c:425:resolve_anonfd_simple] 0-server: inode for the gfid (ea84b493-b049-487a-825b-cb4bece296d7) is not found. anonymous fd creation failed
[2015-03-17 11:56:04.172063] W [server-resolve.c:425:resolve_anonfd_simple] 0-server: inode for the gfid (ea84b493-b049-487a-825b-cb4bece296d7) is not found. anonymous fd creation failed
[2015-03-17 11:56:04.172489] W [server-resolve.c:425:resolve_anonfd_simple] 0-server: inode for the gfid (b8bb35ff-1641-49c8-8ce2-b07f35fd4ef7) is not found. anonymous fd creation failed
[2015-03-17 11:56:04.172567] W [server-resolve.c:425:resolve_anonfd_simple] 0-server: inode for the gfid (b8bb35ff-1641-49c8-8ce2-b07f35fd4ef7) is not found. anonymous fd creation failed

Version-Release number of selected component (if applicable):
=============================================================
[root@vertigo tmp]# gluster --version
glusterfs 3.7dev built on Mar 12 2015 01:40:59
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@vertigo tmp]# 

How reproducible:
=================
Tried once


Steps to Reproduce:
1. Create a 1x(8+4) disperse volume and nfs/fuse mount on the client
2. Untar linux kernel images in parallel of about 20 instances
3. Bring down 4 bricks for some time and bring them back up with "gluster volume start force"
4. Trigger full heal from server
5. On the client do a 'du -sh  ." on the untarred directory. (Not sure if this has to do anything with log getting filled up)

Actual results:


Expected results:


Additional info:
================
sosreports of server and clients will be attached.

--- Additional comment from Ashish Pandey on 2015-05-12 07:58:04 EDT ---

I tried to reproduce the issue with latest code but could not see such  ("anonymous fd creation failed") logs and failures.
During these steps logs were getting created but I could not find anything which may cause logs filled.
Please check with latest code and see if you still see the above logs or any other problematic logs.

--- Additional comment from Anand Avati on 2015-05-19 07:37:53 EDT ---

REVIEW: http://review.gluster.org/10829 (Changing log level from Warning to Debug) posted (#1) for review on master by Ashish Pandey (aspandey)

--- Additional comment from Anand Avati on 2015-05-28 22:36:13 EDT ---

COMMIT: http://review.gluster.org/10829 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit e6934debda3a67b5666f1aa30b33c5267de49d84
Author: Ashish Pandey <aspandey>
Date:   Tue May 19 17:04:55 2015 +0530

    Changing log level from Warning to Debug
    
    Change-Id: Ic4306ebb5a58c2ca310e973bb562be1ca3e4885f
    BUG: 1202758
    Signed-off-by: Ashish Pandey <aspandey>
    Reviewed-on: http://review.gluster.org/10829
    Tested-by: NetBSD Build System
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 1 Anand Avati 2015-06-04 08:32:31 UTC
REVIEW: http://review.gluster.org/10998 (Changing log level from Warning to Debug) posted (#3) for review on release-3.7 by Ashish Pandey (aspandey)

Comment 2 Niels de Vos 2015-06-20 09:49:52 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.2, please reopen this bug report.

glusterfs-3.7.2 has been announced on the Gluster Packaging mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/packaging/2015-June/000006.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.