Bug 1447020

Summary:	Nfs server goes down frequently on one of the glusterfs servers
Product:	[Community] GlusterFS	Reporter:	Aditya C S <aditya.gnu>
Component:	glusterd	Assignee:	bugs <bugs>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	3.10	CC:	aditya.gnu, bugs, rkavunga
Target Milestone:	---	Flags:	aditya.gnu: needinfo-
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-05-09 12:22:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Aditya C S 2017-05-01 08:42:47 UTC

I am running a two node cluster. When I try to write something to the mounted location, nfs server goes down on one of the clusters. 

Below are the logs from bricks/*.log

[2017-05-01 08:26:19.415621] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/17/log [No data available]
[2017-05-01 08:26:19.415654] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70832: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/17/log (42bb8306-4187-4626-aac5-1148c3b0f337/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.424695] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/e5/8a/e58a77fa-cad5-4fab-a82e-49350828d230/3/log [No data available]
[2017-05-01 08:26:19.424738] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70856: LOOKUP <gfid:e58a77fa-cad5-4fab-a82e-49350828d230>/3/log (e696c504-4aba-4eaf-a37b-993fd72f4afb/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.436183] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/86/log [No data available]
[2017-05-01 08:26:19.436217] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70880: LOOKUP <gfid:e8031a49-81a7-477a-9f21-aa6a2770a837>/log (e8031a49-81a7-477a-9f21-aa6a2770a837/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.446225] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/16/log [No data available]
[2017-05-01 08:26:19.446258] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70904: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/16/log (50c9101e-d7c7-4787-8ab0-2064922e3387/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.456734] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/85/log [No data available]
[2017-05-01 08:26:19.456767] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70928: LOOKUP <gfid:bd5b1111-c4ce-4ea2-bdd6-29ba14e4051a>/log (bd5b1111-c4ce-4ea2-bdd6-29ba14e4051a/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.467384] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/15/log [No data available]
[2017-05-01 08:26:19.467417] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70952: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/15/log (b7b09ed9-1066-43fb-bbc8-9dc43fd1bbd3/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.478892] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/84/log [No data available]
[2017-05-01 08:26:19.478941] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70976: LOOKUP <gfid:f26fe149-5c72-4dc4-842c-9fb7ef5ac526>/log (f26fe149-5c72-4dc4-842c-9fb7ef5ac526/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.489388] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/14/log [No data available]
[2017-05-01 08:26:19.489421] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 71000: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/14/log (7fdf9e9f-5529-4b51-9d1e-848f068a7cc1/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.498951] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/83/log [No data available]

Thanks

Comment 1 Mohammed Rafi KC 2017-05-02 12:40:46 UTC

To make debugging faster, can you please provide additional logs like nfs.log and bricks log, Also steps to reproduce will surely help.

Comment 2 hari gowtham 2017-05-09 12:22:47 UTC

Hi,

As we don't have enough data to work on the bug, the need info flag was raised. once we get enough data on the bug we will work on it(need info is closed). Until then the bug is being closed. Please feel free to open when the required data is available.