Bug 1205579 - gluster nfs server process was crashed multiple time while mounting volume and starting volume using force option
Summary: gluster nfs server process was crashed multiple time while mounting volume an...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: nfs
Version: mainline
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Niels de Vos
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1196520
TreeView+ depends on / blocked
 
Reported: 2015-03-25 08:59 UTC by Niels de Vos
Modified: 2015-05-14 17:46 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-05-14 17:29:24 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Niels de Vos 2015-03-25 08:59:19 UTC
Description of problem:
=======================
while mounting volume using nfs it was hung and found nfs server process was crashed and core was generated. After that killed some brick process and during start force of that volume found crash again

nfs log snippet:-
[2014-08-09 03:25:04.297623] E [nfs.c:341:nfs_init_versions] 0-nfs: Program  NLM4 registration failed
[2014-08-09 03:25:04.297642] E [nfs.c:1327:init] 0-nfs: Failed to initialize protocols
[2014-08-09 03:25:04.297654] E [xlator.c:423:xlator_init] 0-nfs-server: Initialization of volume 'nfs-server' failed, review your volfile again
[2014-08-09 03:25:04.297684] E [graph.c:292:glusterfs_graph_init] 0-nfs-server: initializing translator failed
[2014-08-09 03:25:04.297698] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed
pending frames:
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11

NLM (of NFS) failed to register with portmapper, so init() routine of NFS xlator returns (-1) to xlator_init(). Call flow:

glusterfsd (NFS process)
  glusterfs_process_volfp() =>
     glusterfs_graph_activate() =>
         glusterfs_graph_init() =>
             xlator_init () => NFS init() routine

But somehow cleanup_and_exit() goes through some race which causes the SEGV. But it clearly says the crash is in the exit code path i.e. NLM failed to register with portmapper and NFS process was about to exit but segfaulted. 


The root cause:
The NLM was not able to register with portmapper which prohibited NFS to start.

Comment 1 Anand Avati 2015-03-25 09:07:31 UTC
REVIEW: http://review.gluster.org/9988 (nfs: do not fail to start when optional RPC-programs fail to register) posted (#1) for review on master by Niels de Vos (ndevos@redhat.com)

Comment 2 Anand Avati 2015-03-25 16:08:50 UTC
REVIEW: http://review.gluster.org/9988 (nfs: do not fail to start when optional RPC-programs fail to register) posted (#2) for review on master by Niels de Vos (ndevos@redhat.com)

Comment 3 Anand Avati 2015-03-26 10:06:01 UTC
COMMIT: http://review.gluster.org/9988 committed in master by Niels de Vos (ndevos@redhat.com) 
------
commit 01d96d97f1bae4f83321285c701b93f2a668ad25
Author: Niels de Vos <ndevos@redhat.com>
Date:   Mon Mar 2 11:14:08 2015 +0100

    nfs: do not fail to start when optional RPC-programs fail to register
    
    Some RPC-programs are not strictly required for the NFS-server. When
    these optional protocols fail to get registered at the portmapper, there
    is no need to fail the starting of the NFS-server.
    
    Required RPC-programs:
     - NFS
     - MNT
    
    Optional RPC-programs:
     - NLM
     - ACL
    
    Change-Id: Ife8ad871cff47554e3f42eb457c76431d0181964
    BUG: 1205579
    Tested-by: Brad Hubbard <bhubbard@redhat.com>
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/9988
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    Reviewed-by: jiffin tony Thottan <jthottan@redhat.com>
    Reviewed-by: soumya k <skoduri@redhat.com>

Comment 4 Niels de Vos 2015-05-14 17:29:24 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 5 Niels de Vos 2015-05-14 17:35:55 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 6 Niels de Vos 2015-05-14 17:38:16 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 7 Niels de Vos 2015-05-14 17:46:38 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.