Bug 1029372

Summary:	Open, Inodelk fops referring to null gfid and null path
Product:	[Community] GlusterFS	Reporter:	Vikhyat Umrao <vumrao>
Component:	logging	Assignee:	Vikhyat Umrao <vumrao>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	3.3.0	CC:	gluster-bugs, ndevos
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.5.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-04-17 11:50:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	916857

Description Vikhyat Umrao 2013-11-12 09:57:40 UTC

Description of problem:
========================
when running dbench on fuse mount for a replicate volume ( 1 x 2 )and one of the bricks goes offline and comes back online, the brick reports the following log messages : 

[2013-03-01 11:34:28.029222] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993888: OPEN (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.031579] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993896: INODELK (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.034041] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993914: INODELK (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.040435] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 993938: FLUSH -2 (--) ==> -1 (No such file or directory)


Version-Release number of selected component (if applicable):
============================================================
[root@luigi:~] Mar-01-2013 12:35:05 $ gluster --version
glusterfs 3.3.0.5rhs built on Feb 19 2013 05:31:58
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.


How reproducible:
==================
Often

Steps to Reproduce:
===================
1. Create replicate volime ( 1 x 2 ) . start the volume

2. Create fuse mount. 

3. run : "  dbench -c /usr/local/share/doc/dbench/loadfiles/client.txt -D <mount_ponit> --stat-check --per-client-results 10 

4. while dbench is running, bring offline one of the bricks.

5. bring back the the brick online 

  
Actual results:
=================
[2013-03-01 11:34:28.029222] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993888: OPEN (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.031579] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993896: INODELK (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.034041] I [server3_1-fops.c:252:server_inodelk_cbk] 0-vol-rep-1-server: 993914: INODELK (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.040435] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 993938: FLUSH -2 (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.040546] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993939: OPEN (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.041595] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993952: OPEN (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.043890] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 993964: OPEN (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.045784] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 993975: FLUSH -2 (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.076810] I [server3_1-fops.c:1541:server_open_cbk] 0-vol-rep-1-server: 994130: OPEN (null) (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.078079] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 994142: FLUSH -2 (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.297736] I [server3_1-fops.c:1338:server_flush_cbk] 0-vol-rep-1-server: 994954: FLUSH -2 (--) ==> -1 (No such file or directory)
[2013-03-01 11:34:28.329278] E [posix.c:361:posix_setattr] 0-vol-rep-1-posix: setattr (lstat) on /home/brick0/.glusterfs/56/cb/56cb58de-5d1d-4d72-a264-d93f74b40345 fail
ed: No such file or directory

Expected Output:-
=================
OPEN, INODELK, FLUSH shouldn't refer to null gfid/path

Comment 1 Anand Avati 2013-11-12 13:18:54 UTC

REVIEW: http://review.gluster.org/6252 (server/rpc: bricks goes offline and comes back online, with lots of "No such file or directory" log messages) posted (#1) for review on master by Vikhyat Umrao (vumrao)

Comment 2 Anand Avati 2013-11-13 07:38:13 UTC

COMMIT: http://review.gluster.org/6252 committed in master by Anand Avati (avati) 
------
commit eeac099cf405849f07989c410968e77f95acdc77
Author: Vikhyat Umrao <vumrao>
Date:   Tue Nov 12 18:11:23 2013 +0530

    server/rpc: bricks goes offline and comes back online, with lots of "No such file or directory" log messages
    
    Problem:
    Messages were getting logged very frequently at log level INFO.
    
    [2013-03-01 11:34:28.029222] I [server3_1-fops.c:1541:server_open_cbk] vol-server: 993888: OPEN (null) (--) ==> -1 (No such file or directory)
    [2013-03-01 11:34:28.031579] I [server3_1-fops.c:252:server_inodelk_cbk] vol-server: 993896: INODELK (null) (--) ==> -1 (No such file or directory)
    [2013-03-01 11:34:28.034041] I [server3_1-fops.c:252:server_inodelk_cbk] vol-server: 993914: INODELK (null) (--) ==> -1 (No such file or directory)
    [2013-03-01 11:34:28.040435] I [server3_1-fops.c:1338:server_flush_cbk] vol-server: 993938: FLUSH -2 (--) ==> -1 (No such file or directory)
    
    Solution:
    Moved them to DEBUG log level if error number equlas to ENOENT else to ERROR log level.
    It will help in decreasing the size of log files at INFO log level.
    For server_open_cbk and for some other functions we already have a patch, below is the URL for it.
    URL- http://review.gluster.org/6241
    This patch solves logging problem for functions server_inodelk_cbk and server_flush_cbk.
    
    Change-Id: I57372e851371e466f1674726015e28378b826f5f
    BUG: 1029372
    Signed-off-by: Vikhyat Umrao<vumrao>
    Reviewed-on: http://review.gluster.org/6252
    Reviewed-by: Niels de Vos <ndevos>
    Tested-by: Gluster Build System <jenkins.com>

Comment 3 Niels de Vos 2014-04-17 11:50:36 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report.

glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user