Bug 1375137

Summary: Ganesha crashes with segfault during export/unexport of volume.
Product: [Community] GlusterFS Reporter: Shashank Raj <sraj>
Component: ganesha-nfsAssignee: Soumya Koduri <skoduri>
Status: CLOSED WORKSFORME QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.8CC: bugs, jthottan, kkeithle, ndevos, sarumuga, sashinde, skoduri, storage-qa-internal
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-27 11:43:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shashank Raj 2016-09-12 09:21:37 UTC
Description of problem:

Ganesha crashes with segfault during export/unexport of volume.

Version-Release number of selected component (if applicable):

[root@dhcp43-116 ~]# rpm -qa|grep glusterfs
glusterfs-geo-replication-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-api-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-fuse-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-server-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-libs-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-client-xlators-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-ganesha-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-cli-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-debuginfo-3.8.3-0.6.git7956718.el7.centos.x86_64
glusterfs-3.8.3-0.6.git7956718.el7.centos.x86_64

[root@dhcp43-116 ~]# rpm -qa|grep ganesha
nfs-ganesha-gluster-next.20160827.7641daf-1.el7.centos.x86_64
glusterfs-ganesha-3.8.3-0.6.git7956718.el7.centos.x86_64
nfs-ganesha-debuginfo-next.20160827.7641daf-1.el7.centos.x86_64
nfs-ganesha-next.20160827.7641daf-1.el7.centos.x86_64

How reproducible:

twice

Steps to Reproduce:
1.Setup nfs-ganesha cluster.
2.Try export/unexport of volume via ganesha multiple times.
3.Observe that ganesha crashes while exporting the volume sometimes with below bt:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f52429a3700 (LWP 16497)]
cih_fhcache_inline_lookup (tree=0x7f52fa348fe8, key=<synthetic pointer>)
    at /usr/src/debug/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_hash.h:179
179			res = cih_fh_cmpf(node, key);
(gdb) bt
#0  cih_fhcache_inline_lookup (tree=0x7f52fa348fe8, key=<synthetic pointer>)
    at /usr/src/debug/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_hash.h:179
#1  cih_get_by_key_latch (key=0x7f52429a17f0, latch=latch@entry=0x7f52429a1740, 
    flags=flags@entry=5, line=623, 
    func=0x7f52f90f1160 <__func__.25539> "mdcache_find_keyed")
    at /usr/src/debug/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_hash.h:322
#2  0x00007f52f90be4b8 in mdcache_find_keyed (key=<optimized out>, 
    entry=0x7f52429a1888)
    at /usr/src/debug/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:621
#3  0x00007f52f90be700 in mdcache_new_entry (export=export@entry=0x7f523c035840, 
    sub_handle=0x7f523cbfdad8, attrs_in=attrs_in@entry=0x7f52429a1890, 
    attrs_out=attrs_out@entry=0x0, new_directory=new_directory@entry=false, 
    entry=entry@entry=0x7f52429a1888, state=state@entry=0x0)
    at /usr/src/debug/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:347
#4  0x00007f52f90bb244 in mdcache_lookup_path (exp_hdl=0x7f523c035840, 
    path=0x7f523c001740 "/v1", handle=0x7f52429a19e8, attrs_out=0x0)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1578
#5  0x00007f52f908fd7d in init_export_root (export=export@entry=0x7f523cc1b2f8)
    at /usr/src/debug/nfs-ganesha/src/support/exports.c:2161
#6  0x00007f52f9090df9 in export_commit_common (self_struct=0x7f523cc1b2f8, 
    err_type=0x7f52429a21c0, commit_type=add_export, link_mem=<optimized out>, 
    node=<optimized out>)
    at /usr/src/debug/nfs-ganesha/src/support/exports.c:1256
#7  0x00007f52f90c7971 in proc_block (node=<optimized out>, 
    item=<optimized out>, link_mem=<optimized out>, err_type=<optimized out>)
    at /usr/src/debug/nfs-ganesha/src/config_parsing/config_parsing.c:1344
#8  0x00007f52f90c8129 in load_config_from_node (tree_node=0x7f523c0085f0, 
    conf_blk=0x7f52f9321720 <add_export_param>, param=param@entry=0x0, 
    unique=unique@entry=false, err_type=err_type@entry=0x7f52429a21c0)
    at /usr/src/debug/nfs-ganesha/src/config_parsing/config_parsing.c:1843
#9  0x00007f52f909e7a7 in gsh_export_addexport (args=<optimized out>, 
    reply=0x7f52fa41db70, error=0x7f52429a22e0)
    at /usr/src/debug/nfs-ganesha/src/support/export_mgr.c:967
#10 0x00007f52f90c2d69 in dbus_message_entrypoint (conn=0x7f52fa41d940, 
    msg=0x7f52fa41dd20, user_data=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/nfs-ganesha/src/dbus/dbus_server.c:512
#11 0x00007f52f894ec86 in _dbus_object_tree_dispatch_and_unlock ()
   from /lib64/libdbus-1.so.3
#12 0x00007f52f8940e49 in dbus_connection_dispatch () from /lib64/libdbus-1.so.3
#13 0x00007f52f89410e2 in _dbus_connection_read_write_dispatch ()
   from /lib64/libdbus-1.so.3
#14 0x00007f52f90c3de0 in gsh_dbus_thread (arg=<optimized out>)
    at /usr/src/debug/nfs-ganesha/src/dbus/dbus_server.c:737
#15 0x00007f52f756adc5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f52f6c381cd in clone () from /lib64/libc.so.6
(gdb) 

Actual results:

Ganesha crashes with segfault during export/unexport of volume.

Expected results:

There should not be any crashes.

Additional info:

sosreport and logs will be attached.

Comment 1 Shashank Raj 2016-09-16 07:15:58 UTC
Tested with the private build provided as below:

[root@dhcp43-116 exports]# rpm -qa|grep ganesha
glusterfs-ganesha-3.8.3-0.6.git7956718.el7.centos.x86_64
nfs-ganesha-gluster-2.4-0.rc4.el7.centos.x86_64
nfs-ganesha-debuginfo-2.4-0.rc4.el7.centos.x86_64
nfs-ganesha-2.4-0.rc4.el7.centos.x86_64

and during basic testing i don't see any ganesha crash/segfault with multiple export/unexport. Will keep continuing with the testing and update the bug if we see any issues later.

Comment 2 Shashank Raj 2016-09-27 11:43:44 UTC
Since this bug was not seen with latest upstream testing, closing it.