Bug 1746324 - [Ganesha] Ganesha crashed on two of the node while exporting the volume
Summary: [Ganesha] Ganesha crashed on two of the node while exporting the volume
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.5.z Batch Update 4
Assignee: Soumya Koduri
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On: 1745389 1748448 1800703
Blocks: 1725716
TreeView+ depends on / blocked
 
Reported: 2019-08-28 07:44 UTC by Manisha Saini
Modified: 2021-04-29 07:20 UTC (History)
16 users (show)

Fixed In Version: glusterfs-6.0-30.1
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-29 07:20:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:1462 0 None None None 2021-04-29 07:20:53 UTC

Description Manisha Saini 2019-08-28 07:44:49 UTC
Description of problem:
========================

Ganesha got crashed on one of the node while exporting volume

---------------------------
Reading symbols from /usr/bin/ganesha.nfsd...Reading symbols from /usr/lib/debug/usr/bin/ganesha.nfsd.debug...done.
done.
Missing separate debuginfo for /lib64/libntirpc.so.1.7
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ba/f482e25a6a8dbfb3665ccc5c81f3bef51b5b30
Missing separate debuginfo for /lib64/libwbclient.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b4/1852bdd635e26adba49a0e2f4e2f6e0165e27b
Missing separate debuginfo for 
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ba/3372036854a6399a5260547cce7841b54ad536
Missing separate debuginfo for /lib64/libntirpc.so.1.7
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ba/f482e25a6a8dbfb3665ccc5c81f3bef51b5b30.debug
Missing separate debuginfo for /lib64/libwbclient.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b4/1852bdd635e26adba49a0e2f4e2f6e0165e27b.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3a901ec435 in syncop_stat (subvol=subvol@entry=0x7f3a700c2100, loc=loc@entry=0x7f3a74f2cca0, stbuf=stbuf@entry=0x7f3a74f2cce0, 
    xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:2423
2423	    SYNCOP(subvol, (&args), syncop_fstat_cbk, subvol->fops->stat, loc,
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-13.el7_6.x86_64 elfutils-libelf-0.176-2.el7.x86_64 elfutils-libs-0.176-2.el7.x86_64 glibc-2.17-292.el7.x86_64 gssproxy-0.7.0-26.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-61.el7.x86_64 libcap-2.22-10.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libuuid-2.23.2-61.el7.x86_64 libwbclient-4.9.1-6.el7.x86_64 lz4-1.7.5-3.el7.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pcre-8.32-17.el7.x86_64 samba-client-libs-4.9.1-6.el7.x86_64 systemd-libs-219-67.el7_7.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  0x00007f3a901ec435 in syncop_stat (subvol=subvol@entry=0x7f3a700c2100, loc=loc@entry=0x7f3a74f2cca0, stbuf=stbuf@entry=0x7f3a74f2cce0, 
    xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:2423
#1  0x00007f3a904aedab in pub_glfs_h_create_from_handle (fs=0x7f3a340058c0, handle=handle@entry=0x7f3a74f2ce70 "", len=len@entry=16, 
    stat=stat@entry=0x7f3a74f2cde0) at glfs-handleops.c:1339
#2  0x00007f3a906bf30e in create_handle (export_pub=0x7f3a34005660, fh_desc=<optimized out>, pub_handle=0x7f3a74f2cf48, attrs_out=0x7f3a74f2cf70)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/FSAL/FSAL_GLUSTER/export.c:239
#3  0x000055864ab7ea1f in mdcache_locate_host (fh_desc=0x7f3a74f2d160, export=export@entry=0x7f3a34005340, entry=entry@entry=0x7f3a74f2d0f0, 
    attrs_out=attrs_out@entry=0x0) at /usr/src/debug/nfs-ganesha-2.7.3/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1055
#4  0x000055864ab77caa in mdcache_create_handle (exp_hdl=0x7f3a34005340, fh_desc=<optimized out>, handle=0x7f3a74f2d158, attrs_out=0x0)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1578
#5  0x000055864aac6e92 in nfs4_mds_putfh (data=data@entry=0x7f3a74f2d720) at /usr/src/debug/nfs-ganesha-2.7.3/src/Protocols/NFS/nfs4_op_putfh.c:211
#6  0x000055864aac73c8 in nfs4_op_putfh (op=0x7f3a500019b0, data=0x7f3a74f2d720, resp=0x7f3a50002d60)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/Protocols/NFS/nfs4_op_putfh.c:281
#7  0x000055864aab5703 in nfs4_Compound (arg=<optimized out>, req=<optimized out>, res=0x7f3a50001c90)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/Protocols/NFS/nfs4_Compound.c:942
#8  0x000055864aaa8b1f in nfs_rpc_process_request (reqdata=0x7f3a50001170) at /usr/src/debug/nfs-ganesha-2.7.3/src/MainNFSD/nfs_worker_thread.c:1328
#9  0x000055864aaa7fca in nfs_rpc_decode_request (xprt=0x7f3a6c000c20, xdrs=0x7f3a50000f90)
    at /usr/src/debug/nfs-ganesha-2.7.3/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1345
#10 0x00007f3a974e562d in svc_rqst_xprt_task () from /lib64/libntirpc.so.1.7
#11 0x00007f3a974e5b6a in svc_rqst_run_task () from /lib64/libntirpc.so.1.7
#12 0x00007f3a974edc0b in work_pool_thread () from /lib64/libntirpc.so.1.7
#13 0x00007f3a95883ea5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f3a9518e8cd in clone () from /lib64/libc.so.6
------------------------


Version-Release number of selected component (if applicable):
=======================================================

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.7.3-7.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.7.3-7.el7rhgs.x86_64
glusterfs-ganesha-6.0-11.el7rhgs.x86_64
nfs-ganesha-2.7.3-7.el7rhgs.x86_64



How reproducible:
=================
1/1

Steps to Reproduce:
===================
1. Created 4 node ganesha cluster
2. Created distributed-replicated volume (mani1). Exported the volume via ganesha
3. Mounted the volume on 3 clients via v4.1
4. Ran some IO's for couple of days
5. Unexported the volume.Volume was successfully unexported from all the nodes
6. Created new volume Distributed-Disperse (mani).
7. Exported the volume via ganesha ( gluster v set mani ganesha.enable on)

Actual results:
===============
Ganesha got crashed on one of the node


Expected results:
================
Ganesha should not crash


Additional info:

Comment 8 Manisha Saini 2019-10-03 09:06:44 UTC
Tried reproducing this issue with 

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.7.3-8.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.7.3-7.el7rhgs.x86_64
glusterfs-ganesha-6.0-15.el7rhgs.x86_64
nfs-ganesha-2.7.3-8.el7rhgs.x86_64


I was unable to reproduce this issue with steps mentioned in comment #0 in 2 attempts.
Will update this bug if I could hit this issue in later regression cycle

Comment 30 errata-xmlrpc 2021-04-29 07:20:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1462


Note You need to log in before you can comment on or make changes to this bug.