Bug 1492995 - [Ganesha] : Ganesha crashed while bringing up the Ganesha service on crashed nodes.
Summary: [Ganesha] : Ganesha crashed while bringing up the Ganesha service on crashed ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: nfs-ganesha
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: RHGS 3.4.0
Assignee: Soumya Koduri
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks: 1503134
TreeView+ depends on / blocked
 
Reported: 2017-09-19 06:42 UTC by Ambarish
Modified: 2018-09-24 11:52 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-04 06:53:36 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2610 0 None None None 2018-09-04 06:54:41 UTC

Description Ambarish 2017-09-19 06:42:01 UTC
Description of problem:
-----------------------

I hit this while repro'ing https://bugzilla.redhat.com/show_bug.cgi?id=1481040 in QE labs.

Ganesha crashed on 3 of my nodes with BT mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1492695.


When I did a service start if NFS-Ganesha on the nodes it crashed,it crashed again and dumped a core with the following BT :

(gdb) bt
#0  0x00007f7f323ecad5 in list_del_init (old=0x7f7f4435f858) at list.h:87
#1  __iobuf_arena_destroy_iobufs (iobuf_arena=iobuf_arena@entry=0x7f7f4435f4d0) at iobuf.c:132
#2  0x00007f7f323ecc3a in __iobuf_arena_destroy (iobuf_pool=<optimized out>, iobuf_arena=0x7f7f4435f4d0) at iobuf.c:153
#3  0x00007f7f323ed2b7 in iobuf_pool_destroy (iobuf_pool=0x7f7f44399ae0) at iobuf.c:331
#4  0x00007f7f326967ce in glusterfs_ctx_destroy (ctx=0x7f7f4435f9e0) at glfs.c:1055
#5  pub_glfs_fini (fs=fs@entry=0x7f7f44f59a60) at glfs.c:1232
#6  0x00007f7f32ac3d9b in glusterfs_create_export (fsal_hdl=<optimized out>, parse_node=<optimized out>, err_type=<optimized out>, up_ops=0x7f7f377cb800 <fsal_up_top>)
    at /usr/src/debug/nfs-ganesha-2.4.1/src/FSAL/FSAL_GLUSTER/export.c:772
#7  0x00007f7f375418e0 in fsal_cfg_commit (node=0x7f7f38bd4c50, link_mem=0x7f7f4435f388, self_struct=<optimized out>, err_type=0x7ffe149180a0) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/exports.c:751
#8  0x00007f7f3757a848 in proc_block (node=<optimized out>, item=<optimized out>, link_mem=<optimized out>, err_type=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1337
#9  0x00007f7f37579cc0 in do_block_load (err_type=<optimized out>, param_struct=<optimized out>, relax=<optimized out>, params=<optimized out>, blk=<optimized out>)
    at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1195
#10 proc_block (node=<optimized out>, item=<optimized out>, link_mem=<optimized out>, err_type=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1321
#11 0x00007f7f3757b130 in load_config_from_parse (config=config@entry=0x7f7f38bbd350 "Pӻ8\177\177", conf_blk=conf_blk@entry=0x7f7f377cf2a0 <export_param>, param=param@entry=0x0, unique=unique@entry=false, 
    err_type=err_type@entry=0x7ffe149180a0) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1908
#12 0x00007f7f3754215b in ReadExports (in_config=0x7f7f38bbd350 "Pӻ8\177\177", err_type=err_type@entry=0x7ffe149180a0) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/exports.c:1970
#13 0x00007f7f37491291 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/MainNFSD/nfs_main.c:454
(gdb) q


It crashed on all the 3 nodes with the same BT.
 

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

nfs-ganesha-2.4.1-11.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-18.4.el7rhgs.x86_64



How reproducible:
-----------------

1/1

Comment 4 Ambarish 2017-09-20 13:12:12 UTC
Similar(but not same) crash on another node while restarting Ganesha :


(gdb) bt
#0  0x00007f6d7cccbad5 in list_del_init (old=0x7f6d889bbf78) at list.h:87
#1  __iobuf_arena_destroy_iobufs (iobuf_arena=iobuf_arena@entry=0x7f6d8898d020) at iobuf.c:132
#2  0x00007f6d7cccbc3a in __iobuf_arena_destroy (iobuf_pool=<optimized out>, iobuf_arena=0x7f6d8898d020) at iobuf.c:153
#3  0x00007f6d7cccc2b7 in iobuf_pool_destroy (iobuf_pool=0x7f6d889c7750) at iobuf.c:331
#4  0x00007f6d7cf757ce in glusterfs_ctx_destroy (ctx=0x7f6d8898d650) at glfs.c:1055
#5  pub_glfs_fini (fs=0x7f6d895876d0) at glfs.c:1232
#6  0x00007f6d7d3a27a1 in export_release (exp_hdl=0x7f6d895a2f70) at /usr/src/debug/nfs-ganesha-2.4.1/src/FSAL/FSAL_GLUSTER/export.c:88
#7  0x00007f6d81e419dd in mdcache_exp_release (exp_hdl=0x7f6d889bb840) at /usr/src/debug/nfs-ganesha-2.4.1/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_export.c:180
#8  0x00007f6d81e2179b in free_export_resources (export=export@entry=0x7f6d8898cdf8) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/exports.c:2068
#9  0x00007f6d81e314a9 in free_export (export=0x7f6d8898cdf8) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/export_mgr.c:252
#10 0x00007f6d81e329b5 in put_gsh_export (export=export@entry=0x7f6d8898cdf8) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/export_mgr.c:632
#11 0x00007f6d81e33a4f in remove_all_exports () at /usr/src/debug/nfs-ganesha-2.4.1/src/support/export_mgr.c:762
#12 0x00007f6d81da6f1a in do_shutdown () at /usr/src/debug/nfs-ganesha-2.4.1/src/MainNFSD/nfs_admin_thread.c:433
#13 admin_thread (UnusedArg=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/MainNFSD/nfs_admin_thread.c:466
#14 0x00007f6d80305dc5 in start_thread (arg=0x7f6939a7e700) at pthread_create.c:308
#15 0x00007f6d7f9d473d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Comment 6 Soumya Koduri 2017-09-21 14:38:02 UTC
Same fixes mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1492695#c9 address this issue as well.

Comment 11 Manisha Saini 2018-08-02 07:40:53 UTC
Since this BZ was observed while restarting ganesha service which got crashed while performing steps in  BZ  1492695

BZ  1492695 has been now verified.Since the reproducer steps are same for both these bugs,moving this BZ also to verified state.

# rpm -qa | grep ganesha
nfs-ganesha-2.5.5-9.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.5.5-9.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-15.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-9.el7rhgs.x86_64

Comment 13 errata-xmlrpc 2018-09-04 06:53:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2610


Note You need to log in before you can comment on or make changes to this bug.