Bug 1447020 - Nfs server goes down frequently on one of the glusterfs servers
Summary: Nfs server goes down frequently on one of the glusterfs servers
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: 3.10
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-01 08:42 UTC by Aditya C S
Modified: 2021-09-22 06:15 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-05-09 12:22:47 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:
aditya.gnu: needinfo-

Attachments (Terms of Use)

Description Aditya C S 2017-05-01 08:42:47 UTC 
I am running a two node cluster. When I try to write something to the mounted location, nfs server goes down on one of the clusters. 

Below are the logs from bricks/*.log

[2017-05-01 08:26:19.415621] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/17/log [No data available]
[2017-05-01 08:26:19.415654] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70832: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/17/log (42bb8306-4187-4626-aac5-1148c3b0f337/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.424695] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/e5/8a/e58a77fa-cad5-4fab-a82e-49350828d230/3/log [No data available]
[2017-05-01 08:26:19.424738] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70856: LOOKUP <gfid:e58a77fa-cad5-4fab-a82e-49350828d230>/3/log (e696c504-4aba-4eaf-a37b-993fd72f4afb/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.436183] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/86/log [No data available]
[2017-05-01 08:26:19.436217] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70880: LOOKUP <gfid:e8031a49-81a7-477a-9f21-aa6a2770a837>/log (e8031a49-81a7-477a-9f21-aa6a2770a837/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.446225] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/16/log [No data available]
[2017-05-01 08:26:19.446258] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70904: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/16/log (50c9101e-d7c7-4787-8ab0-2064922e3387/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.456734] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/85/log [No data available]
[2017-05-01 08:26:19.456767] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70928: LOOKUP <gfid:bd5b1111-c4ce-4ea2-bdd6-29ba14e4051a>/log (bd5b1111-c4ce-4ea2-bdd6-29ba14e4051a/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.467384] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/15/log [No data available]
[2017-05-01 08:26:19.467417] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70952: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/15/log (b7b09ed9-1066-43fb-bbc8-9dc43fd1bbd3/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.478892] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/84/log [No data available]
[2017-05-01 08:26:19.478941] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 70976: LOOKUP <gfid:f26fe149-5c72-4dc4-842c-9fb7ef5ac526>/log (f26fe149-5c72-4dc4-842c-9fb7ef5ac526/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.489388] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/8c/b1/8cb16f84-4be2-47a7-af04-0bd21d8ffe86/14/log [No data available]
[2017-05-01 08:26:19.489421] E [MSGID: 115050] [server-rpc-fops.c:158:server_lookup_cbk] 0-nfs-volume-server: 71000: LOOKUP <gfid:c4d7c099-10cd-4418-b0cd-f68ddb9b33ec>/builds/14/log (7fdf9e9f-5529-4b51-9d1e-848f068a7cc1/log), client: gluster1-10475-2017/05/01-07:45:51:783693-nfs-volume-client-0-0-0, error-xlator: nfs-volume-posix [No data available]
[2017-05-01 08:26:19.498951] E [MSGID: 113002] [posix.c:253:posix_lookup] 0-nfs-volume-posix: buf->ia_gfid is null for /home/devstorage/.glusterfs/75/33/75333acd-9b7e-4953-adef-72e6db45a174/83/log [No data available]

Thanks
Comment 1 Mohammed Rafi KC 2017-05-02 12:40:46 UTC 
To make debugging faster, can you please provide additional logs like nfs.log and bricks log, Also steps to reproduce will surely help.
Comment 2 hari gowtham 2017-05-09 12:22:47 UTC 
Hi,

As we don't have enough data to work on the bug, the need info flag was raised. once we get enough data on the bug we will work on it(need info is closed). Until then the bug is being closed. Please feel free to open when the required data is available.
Note You need to log in before you can comment on or make changes to this bug.