Bug 1473280 - [Ganesha] : Ganesha crashed on all the nodes during ganesha-disable; nfs-ganesha disable got timed out on CLI.
Summary: [Ganesha] : Ganesha crashed on all the nodes during ganesha-disable; nfs-gane...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: nfs-ganesha
Version: rhgs-3.3
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ---
: ---
Assignee: Kaleb KEITHLEY
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks: 1417153
TreeView+ depends on / blocked
 
Reported: 2017-07-20 11:59 UTC by Ambarish
Modified: 2019-05-20 12:41 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
As part of ‘gluster nfs-ganesha disable’ command, NFS-Ganesha service is stopped. In case of pre exported entries, NFS-Ganesha may enter FAILED state. Workaround: Restart the NFS-Ganesha process after failure and rerun the ‘gluster nfs-ganesha disable’ command.
Clone Of:
Environment:
Last Closed: 2019-05-20 12:41:15 UTC
Embargoed:

Attachments (Terms of Use)

Description Ambarish 2017-07-20 11:59:01 UTC 
Description of problem:
-----------------------

4 node cluster.

Enabled Ganesha.USed it for a while,pumped lots of IO from v3 and v4.

Disabled Ganesha as a part of teardown

Ganesha crashed on all the nodes and dumped the following core :


<BT>

(gdb) bt
#0  0x00007f74571481f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f74571498e8 in __GI_abort () at abort.c:90
#2  0x00007f7457187f47 in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x7f7457294608 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007f745718f619 in malloc_printerr (ar_ptr=0x7f74574cf760 <main_arena>, ptr=<optimized out>, 
    str=0x7f7457294710 "double free or corruption (out)", action=3) at malloc.c:5023
#4  _int_free (av=0x7f74574cf760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3845
#5  0x00005614e87ac1e1 in shutdown_export (export=0x7f73a0003300)
    at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/fsal_destroyer.c:152
#6  destroy_fsals () at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/fsal_destroyer.c:194
#7  0x00005614e87d3c7f in do_shutdown () at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_admin_thread.c:446
#8  admin_thread (UnusedArg=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_admin_thread.c:466
#9  0x00007f7457b3de25 in start_thread (arg=0x7f73a7ef7700) at pthread_create.c:308
#10 0x00007f745720b34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) 

<BT>

Version-Release number of selected component (if applicable):
------------------------------------------------------------

glusterfs-ganesha-3.8.4-34.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-16.el7rhgs.x86_64

How reproducible:
-----------------

Reporting the first occurence.


Additional info:
----------------

gluster nfs-ganesha disable got timed out was well as a result.
Comment 2 Ambarish 2017-07-20 12:01:11 UTC 
<snip>

[root@gqas013 tmp]# gluster nfs-ganesha disable
Disabling NFS-Ganesha will tear down entire ganesha cluster across the trusted pool. Do you still want to continue?
 (y/n) y
This will take a few minutes to complete. Please wait ..

Error : Request timed out
[root@gqas013 tmp]# 

</snip>
Comment 10 Daniel Gryniewicz 2017-07-21 13:47:21 UTC 
I looked at the code, and all that happens is that systemd/upstart is used to shutdown ganesha.  No extra steps, like dbus unexport, seem to happen.  Have I read the code correctly?
Note You need to log in before you can comment on or make changes to this bug.