Bug 2419138
| Summary: | [NFS-Ganesha] NFS crashed and dumped core (deconstruct_handle) while performing Cluster QOS tests with NFS service restart | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Arunkumar Ravindranath <aravindr> |
| Component: | NFS-Ganesha | Assignee: | Sachin Punadikar <spunadik> |
| NFS-Ganesha sub component: | Ceph | QA Contact: | Manisha Saini <msaini> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | urgent | ||
| Priority: | unspecified | CC: | cephqe-warriors, ffilz, giboye6264, kkeithle, nchillar, ngangadh, spunadik, tserlin, vereddy |
| Version: | 9.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 9.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | nfs-ganesha-7.0-0.6.11 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2026-01-29 07:04:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Reading symbols from /usr/lib/debug/usr/bin/ganesha.nfsd-7.0-0.6.10.el9cp.x86_64.debug... https://blox-fruits.io Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2026:1536 |
Description of problem: We are consistently hitting this coredump on this test: Qos PerShare_PerClient-combined_bw enablement on Cluster level with nfs with service restart Version-Release number of selected component (if applicable): # ceph version ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo) # ceph versions { "mon": { "ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo)": 3 }, "mgr": { "ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo)": 18 }, "mds": { "ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo)": 2 }, "rgw": { "ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo)": 2 }, "overall": { "ceph version 20.1.0-118.el9cp (9bc7fcf9239e20437535a6bba6f48d2500abe8e8) tentacle (rc - RelWithDebInfo)": 27 } } nfs ganesha versions # rpm -qa | grep nfs libnfsidmap-2.5.4-38.el9.x86_64 nfs-utils-2.5.4-38.el9.x86_64 nfs-ganesha-selinux-7.0-0.6.10.el9cp.noarch nfs-ganesha-7.0-0.6.10.el9cp.x86_64 nfs-ganesha-ceph-7.0-0.6.10.el9cp.x86_64 nfs-ganesha-rados-grace-7.0-0.6.10.el9cp.x86_64 nfs-ganesha-rados-urls-7.0-0.6.10.el9cp.x86_64 nfs-ganesha-rgw-7.0-0.6.10.el9cp.x86_64 nfs-ganesha-utils-7.0-0.6.10.el9cp.x86_64 How reproducible: 100% Steps to Reproduce: 1. Setup NFS cluster 2. Enable Qos in cluster level for PerShare_PerClient 3. Restart the cluster and verify if the QOS data persists after restart. Actual results: No Coredump expected Also, limit is not applied Expected results: Core-dump found Additional info: 1. Found during automation run 2. GDB backtace: # gdb /usr/bin/ganesha.nfsd 'core.ganesha\\x2enfsd.0.4025cd670a774ef0a251975745a77a1f.88000.1764925251000000' GNU gdb (Red Hat Enterprise Linux) 16.3-2.el9 Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/ganesha.nfsd... Reading symbols from /usr/lib/debug/usr/bin/ganesha.nfsd-7.0-0.6.10.el9cp.x86_64.debug... [New LWP 53] [New LWP 39] [New LWP 66] [New LWP 73] [New LWP 2] [New LWP 41] [New LWP 42] [New LWP 55] [New LWP 58] [New LWP 59] [New LWP 60] [New LWP 69] [New LWP 74] [New LWP 65] [New LWP 72] [New LWP 71] [New LWP 40] [New LWP 56] [New LWP 57] [New LWP 64] [New LWP 67] [New LWP 68] [New LWP 70] [New LWP 18] [New LWP 38] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/bin/ganesha.nfsd -F -L STDERR -N NIV_EVENT'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ffac0fc1938 in deconstruct_handle (obj=0x7ffab800be30) at /usr/src/debug/nfs-ganesha-7.0-0.6.10.el9cp.x86_64/src/FSAL/FSAL_CEPH/internal.c:127 127 ceph_ll_put(export->cmount, obj->i); [Current thread is 1 (Thread 0x7ffa6d7ea640 (LWP 53))] --Type <RET> for more, q to quit, c to continue without paging-- Missing rpms, try: dnf --enablerepo='*debug*' install libntirpc-debuginfo-7.0-3.el9cp.x86_64 glibc-debuginfo-2.34-231.el9_7.2.x86_64 libwbclient- debuginfo-4.22.4-6.el9_7.x86_64 libnfsidmap-debuginfo-2.5.4-38.el9.x86_64 dbus-libs-debuginfo-1.12.20-8.el9.x86_64 libacl-debuginfo-2.3.1-4.el9.x 86_64 libcap-debuginfo-2.48-10.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 libuuid-debuginfo-2.37.4-21.el9.x86_64 krb5-libs-debuginfo-1.21 .1-8.el9_6.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 grpc-cpp-debuginfo-1.46.7-2.el9cp.x86_64 p rotobuf-debuginfo-3.14.0-16.el9.x86_64 libstdc++-debuginfo-11.5.0-11.el9.x86_64 libgcc-debuginfo-11.5.0-11.el9.x86_64 openssl-libs-debuginfo-3.5. 1-4.el9_7.x86_64 systemd-libs-debuginfo-252-55.el9_7.7.x86_64 libattr-debuginfo-2.5.1-3.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 grp c-debuginfo-1.46.7-2.el9cp.x86_64 abseil-cpp-debuginfo-20211102.0-4.el9cp.x86_64 zlib-debuginfo-1.2.11-40.el9.x86_64 libgcrypt-debuginfo-1.10.0-1 1.el9.x86_64 xz-libs-debuginfo-5.2.5-8.el9_0.x86_64 libzstd-debuginfo-1.5.5-1.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 libselinux-debugin fo-3.6-3.el9.x86_64 c-ares-debuginfo-1.19.1-2.el9_4.x86_64 re2-debuginfo-20211101-4.el9cp.x86_64 libgpg-error-debuginfo-1.42-5.el9.x86_64 pcre2-d ebuginfo-10.40-6.el9.x86_64 libibverbs-debuginfo-57.0-2.el9.x86_64 librdmacm-debuginfo-57.0-2.el9.x86_64 libcurl-minimal-debuginfo-7.76.1-34.el9. x86_64 thrift-debuginfo-0.15.0-3.el9cp.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 lttng-ust-debuginfo-2 .12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 gssproxy-debuginfo-0.8.4-7.el9.x86_64 libicu-debuginfo-67.1-10.el9_6.x86_64 sssd-cl ient-debuginfo-2.9.7-4.el9_7.1.x86_64 (gdb) bt #0 0x00007ffac0fc1938 in deconstruct_handle (obj=0x7ffab800be30) at /usr/src/debug/nfs-ganesha-7.0-0.6.10.el9cp.x86_64/src/FSAL/FSAL_CEPH/internal.c:127 #1 0x00007ffac3e9bad2 in shutdown_handles (fsal=0x7ffac0fcd000 <CephFSM>) at /usr/src/debug/nfs-ganesha-7.0-0.6.10.el9cp.x86_64/src/FSAL/fsal_destroyer.c:67 #2 destroy_fsals () at /usr/src/debug/nfs-ganesha-7.0-0.6.10.el9cp.x86_64/src/FSAL/fsal_destroyer.c:151 #3 0x00007ffac3eb5a86 in do_shutdown () at /usr/src/debug/nfs-ganesha-7.0-0.6.10.el9cp.x86_64/src/MainNFSD/nfs_admin_thread.c:983 #4 admin_thread (UnusedArg=<optimized out>) at /usr/src/debug/nfs-ganesha-7.0-0.6.10.el9cp.x86_64/src/MainNFSD/nfs_admin_thread.c:1007 #5 0x00007ffac3cb02ea in start_thread () from /lib64/libc.so.6 #6 0x00007ffac3d353c0 in clone3 () from /lib64/libc.so.6