Bug 1746324

Summary: [Ganesha] Ganesha crashed on two of the node while exporting the volume
Product: Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: coreAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.5CC: dang, ffilz, grajoria, jahernan, jthottan, mbenjamin, moagrawa, pasik, pprakash, puebele, ravishankar, rhs-bugs, skoduri, storage-qa-internal, vdas, ykaul
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.5.z Batch Update 4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-30.1 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-29 07:20:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1745389, 1748448, 1800703    
Bug Blocks: 1725716    

Description Manisha Saini 2019-08-28 07:44:49 UTC
Description of problem:
========================

Ganesha got crashed on one of the node while exporting volume

---------------------------
Reading symbols from /usr/bin/ganesha.nfsd...Reading symbols from /usr/lib/debug/usr/bin/ganesha.nfsd.debug...done.
done.
Missing separate debuginfo for /lib64/libntirpc.so.1.7
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ba/f482e25a6a8dbfb3665ccc5c81f3bef51b5b30
Missing separate debuginfo for /lib64/libwbclient.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b4/1852bdd635e26adba49a0e2f4e2f6e0165e27b
Missing separate debuginfo for 
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ba/3372036854a6399a5260547cce7841b54ad536
Missing separate debuginfo for /lib64/libntirpc.so.1.7
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ba/f482e25a6a8dbfb3665ccc5c81f3bef51b5b30.debug
Missing separate debuginfo for /lib64/libwbclient.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b4/1852bdd635e26adba49a0e2f4e2f6e0165e27b.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3a901ec435 in syncop_stat (subvol=subvol@entry=0x7f3a700c2100, loc=loc@entry=0x7f3a74f2cca0, stbuf=stbuf@entry=0x7f3a74f2cce0, 
    xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:2423
2423	    SYNCOP(subvol, (&args), syncop_fstat_cbk, subvol->fops->stat, loc,
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-13.el7_6.x86_64 elfutils-libelf-0.176-2.el7.x86_64 elfutils-libs-0.176-2.el7.x86_64 glibc-2.17-292.el7.x86_64 gssproxy-0.7.0-26.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-61.el7.x86_64 libcap-2.22-10.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libuuid-2.23.2-61.el7.x86_64 libwbclient-4.9.1-6.el7.x86_64 lz4-1.7.5-3.el7.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pcre-8.32-17.el7.x86_64 samba-client-libs-4.9.1-6.el7.x86_64 systemd-libs-219-67.el7_7.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  0x00007f3a901ec435 in syncop_stat (subvol=subvol@entry=0x7f3a700c2100, loc=loc@entry=0x7f3a74f2cca0, stbuf=stbuf@entry=0x7f3a74f2cce0, 
    xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:2423
#1  0x00007f3a904aedab in pub_glfs_h_create_from_handle (fs=0x7f3a340058c0, handle=handle@entry=0x7f3a74f2ce70 "", len=len@entry=16, 
    stat=stat@entry=0x7f3a74f2cde0) at glfs-handleops.c:1339
#2  0x00007f3a906bf30e in create_handle (export_pub=0x7f3a34005660, fh_desc=<optimized out>, pub_handle=0x7f3a74f2cf48, attrs_out=0x7f3a74f2cf70)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/FSAL/FSAL_GLUSTER/export.c:239
#3  0x000055864ab7ea1f in mdcache_locate_host (fh_desc=0x7f3a74f2d160, export=export@entry=0x7f3a34005340, entry=entry@entry=0x7f3a74f2d0f0, 
    attrs_out=attrs_out@entry=0x0) at /usr/src/debug/nfs-ganesha-2.7.3/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1055
#4  0x000055864ab77caa in mdcache_create_handle (exp_hdl=0x7f3a34005340, fh_desc=<optimized out>, handle=0x7f3a74f2d158, attrs_out=0x0)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1578
#5  0x000055864aac6e92 in nfs4_mds_putfh (data=data@entry=0x7f3a74f2d720) at /usr/src/debug/nfs-ganesha-2.7.3/src/Protocols/NFS/nfs4_op_putfh.c:211
#6  0x000055864aac73c8 in nfs4_op_putfh (op=0x7f3a500019b0, data=0x7f3a74f2d720, resp=0x7f3a50002d60)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/Protocols/NFS/nfs4_op_putfh.c:281
#7  0x000055864aab5703 in nfs4_Compound (arg=<optimized out>, req=<optimized out>, res=0x7f3a50001c90)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/Protocols/NFS/nfs4_Compound.c:942
#8  0x000055864aaa8b1f in nfs_rpc_process_request (reqdata=0x7f3a50001170) at /usr/src/debug/nfs-ganesha-2.7.3/src/MainNFSD/nfs_worker_thread.c:1328
#9  0x000055864aaa7fca in nfs_rpc_decode_request (xprt=0x7f3a6c000c20, xdrs=0x7f3a50000f90)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1345
#10 0x00007f3a974e562d in svc_rqst_xprt_task () from /lib64/libntirpc.so.1.7
#11 0x00007f3a974e5b6a in svc_rqst_run_task () from /lib64/libntirpc.so.1.7
#12 0x00007f3a974edc0b in work_pool_thread () from /lib64/libntirpc.so.1.7
#13 0x00007f3a95883ea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f3a9518e8cd in clone () from /lib64/libc.so.6
------------------------


Version-Release number of selected component (if applicable):
=======================================================

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.7.3-7.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.7.3-7.el7rhgs.x86_64
glusterfs-ganesha-6.0-11.el7rhgs.x86_64
nfs-ganesha-2.7.3-7.el7rhgs.x86_64



How reproducible:
=================
1/1

Steps to Reproduce:
===================
1. Created 4 node ganesha cluster
2. Created distributed-replicated volume (mani1). Exported the volume via ganesha
3. Mounted the volume on 3 clients via v4.1
4. Ran some IO's for couple of days
5. Unexported the volume.Volume was successfully unexported from all the nodes
6. Created new volume Distributed-Disperse (mani).
7. Exported the volume via ganesha ( gluster v set mani ganesha.enable on)

Actual results:
===============
Ganesha got crashed on one of the node


Expected results:
================
Ganesha should not crash


Additional info:

Comment 8 Manisha Saini 2019-10-03 09:06:44 UTC
Tried reproducing this issue with 

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.7.3-8.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.7.3-7.el7rhgs.x86_64
glusterfs-ganesha-6.0-15.el7rhgs.x86_64
nfs-ganesha-2.7.3-8.el7rhgs.x86_64


I was unable to reproduce this issue with steps mentioned in comment #0 in 2 attempts.
Will update this bug if I could hit this issue in later regression cycle

Comment 30 errata-xmlrpc 2021-04-29 07:20:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1462