Bug 1473280 - [Ganesha] : Ganesha crashed on all the nodes during ganesha-disable; nfs-ganesha disable got timed out on CLI.
[Ganesha] : Ganesha crashed on all the nodes during ganesha-disable; nfs-gane...
Status: ASSIGNED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha (Show other bugs)
3.3
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Kaleb KEITHLEY
Manisha Saini
:
Depends On:
Blocks: 1417153
  Show dependency treegraph
 
Reported: 2017-07-20 07:59 EDT by Ambarish
Modified: 2018-05-11 10:34 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
As part of ‘gluster nfs-ganesha disable’ command, NFS-Ganesha service is stopped. In case of pre exported entries, NFS-Ganesha may enter FAILED state. Workaround: Restart the NFS-Ganesha process after failure and rerun the ‘gluster nfs-ganesha disable’ command.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ambarish 2017-07-20 07:59:01 EDT
Description of problem:
-----------------------

4 node cluster.

Enabled Ganesha.USed it for a while,pumped lots of IO from v3 and v4.

Disabled Ganesha as a part of teardown

Ganesha crashed on all the nodes and dumped the following core :


<BT>

(gdb) bt
#0  0x00007f74571481f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f74571498e8 in __GI_abort () at abort.c:90
#2  0x00007f7457187f47 in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x7f7457294608 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007f745718f619 in malloc_printerr (ar_ptr=0x7f74574cf760 <main_arena>, ptr=<optimized out>, 
    str=0x7f7457294710 "double free or corruption (out)", action=3) at malloc.c:5023
#4  _int_free (av=0x7f74574cf760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3845
#5  0x00005614e87ac1e1 in shutdown_export (export=0x7f73a0003300)
    at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/fsal_destroyer.c:152
#6  destroy_fsals () at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/fsal_destroyer.c:194
#7  0x00005614e87d3c7f in do_shutdown () at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_admin_thread.c:446
#8  admin_thread (UnusedArg=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_admin_thread.c:466
#9  0x00007f7457b3de25 in start_thread (arg=0x7f73a7ef7700) at pthread_create.c:308
#10 0x00007f745720b34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) 

<BT>

Version-Release number of selected component (if applicable):
------------------------------------------------------------

glusterfs-ganesha-3.8.4-34.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-16.el7rhgs.x86_64

How reproducible:
-----------------

Reporting the first occurence.


Additional info:
----------------

gluster nfs-ganesha disable got timed out was well as a result.
Comment 2 Ambarish 2017-07-20 08:01:11 EDT
<snip>

[root@gqas013 tmp]# gluster nfs-ganesha disable
Disabling NFS-Ganesha will tear down entire ganesha cluster across the trusted pool. Do you still want to continue?
 (y/n) y
This will take a few minutes to complete. Please wait ..

Error : Request timed out
[root@gqas013 tmp]# 

</snip>
Comment 10 Daniel Gryniewicz 2017-07-21 09:47:21 EDT
I looked at the code, and all that happens is that systemd/upstart is used to shutdown ganesha.  No extra steps, like dbus unexport, seem to happen.  Have I read the code correctly?

Note You need to log in before you can comment on or make changes to this bug.