Bug 2374668
| Summary: | [8.1z] [NFS-Ganesha][ BYOK] NFS-Ganesha crashing while restarting the ganesha service with 2 NFS exports having diff --kmip_key_id set | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Manisha Saini <msaini> | |
| Component: | NFS-Ganesha | Assignee: | Marcus Watts <mwatts> | |
| Status: | CLOSED UPSTREAM | QA Contact: | ||
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 8.1 | CC: | bkunal, cephqe-warriors, kkeithle | |
| Target Milestone: | --- | Keywords: | External | |
| Target Release: | 8.1z2 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2385946 (view as bug list) | Environment: | ||
| Last Closed: | 2026-03-04 09:53:52 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2385946 | |||
| Bug Blocks: | ||||
This product has been discontinued or is no longer tracked in Red Hat Bugzilla. |
Description of problem: ===================== Enable Encryption while creating the NFS cluster.Create 2 NFS export with different keys.Restart ganesha. On service restart, NFS is crashing --------- Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f30b2b4028e in free () from /lib64/libc.so.6 [Current thread is 1 (Thread 0x7f30b0dd2300 (LWP 2))] Missing separate debuginfos, use: dnf debuginfo-install abseil-cpp-20211102.0-4.el9cp.x86_64 c-ares-1.19.1-2.el9_4.x86_64 dbus-libs-1.12.20-8.el9.x86_64 glibc-2.34-168.el9_6.19.x86_64 grpc-1--Type <RET> for more, q to quit, c to continue without paging-- .46.7-2.el9cp.x86_64 grpc-cpp-1.46.7-2.el9cp.x86_64 keyutils-libs-1.6.3-1.el9.x86_64 krb5-libs-1.21.1-6.el9.x86_64 libacl-2.3.1-4.el9.x86_64 libattr-2.5.1-3.el9.x86_64 libcom_err-1.46.5-7.el9.x86_64 libcurl-minimal-7.76.1-31.el9.x86_64 libgcc-11.5.0-5.el9_5.x86_64 libgcrypt-1.10.0-11.el9.x86_64 libgpg-error-1.42-5.el9.x86_64 libicu-67.1-9.el9.x86_64 libnfsidmap-2.5.4-34.el9.x86_64 libnghttp2-1.43.0-6.el9.x86_64 libnl3-3.11.0-1.el9.x86_64 libntirpc-6.3-3.el9cp.x86_64 librdmacm-54.0-1.el9.x86_64 libselinux-3.6-3.el9.x86_64 libstdc++-11.5.0-5.el9_5.x86_64 libuuid-2.37.4-21.el9.x86_64 libwbclient-4.21.3-5.el9_6.x86_64 lttng-ust-2.12.0-6.el9.x86_64 lz4-libs-1.9.3-5.el9.x86_64 numactl-libs-2.0.19-1.el9.x86_64 openssl-libs-3.2.2-6.el9_5.1.x86_64 pcre2-10.40-6.el9.x86_64 protobuf-3.14.0-16.el9.x86_64 re2-20211101-4.el9cp.x86_64 systemd-libs-252-51.el9.x86_64 thrift-0.15.0-3.el9cp.x86_64 userspace-rcu-0.12.1-6.el9.x86_64 xz-libs-5.2.5-8.el9_0.x86_64 zlib-1.2.11-40.el9.x86_64 (gdb) bt #0 0x00007f30b2b4028e in free () from /lib64/libc.so.6 #1 0x00007f30aff4bb9d in kmip_get_keyvalue (unique_id=<optimized out>, value_out=value_out@entry=0x7ffce6653f08, value_size=value_size@entry=0x7ffce6653f00) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/PLUGIN/kmip/kmip_fscrypt.c:849 #2 0x00007f30aff4c138 in kmip_root_cb_func (cb=0x558642038680, obj=0x5586442f31a8) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/PLUGIN/kmip/kmip_fscrypt.c:889 #3 0x00007f30b2d9c897 in init_export_root (export=export@entry=0x558642039960) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/support/exports.c:3121 #4 0x00007f30b2d9d025 in init_export_cb (exp=0x558642039960, state=0x7ffce66541e0) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/support/exports.c:2948 #5 0x00007f30b2e44373 in foreach_gsh_export.isra.0 (cb=0x7f30b2d9d010 <init_export_cb>, wrlock=wrlock@entry=true, state=state@entry=0x7ffce66541e0) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/support/export_mgr.c:816 #6 0x00007f30b2d3ec8d in exports_pkginit () at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/support/exports.c:2967 #7 nfs_Init (p_start_info=<optimized out>) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/MainNFSD/nfs_init.c:1065 #8 nfs_start (p_start_info=<optimized out>) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/MainNFSD/nfs_init.c:1317 #9 nfs_start (p_start_info=p_start_info@entry=0x55864056c008 <my_nfs_start_info>) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/MainNFSD/nfs_init.c:1283 #10 0x000055864056811f in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/nfs-ganesha-6.5-20.el9cp.x86_64/src/MainNFSD/nfs_main.c:676 (gdb) ------------------ Version-Release number of selected component (if applicable): ---------------------------- # ceph --version ceph version 19.2.1-224.el9cp (7a698d1865dee2d91ba1430045db051c4def6957) squid (stable) # rpm -qa | grep nfs libnfsidmap-2.5.4-34.el9.x86_64 nfs-utils-2.5.4-34.el9.x86_64 nfs-ganesha-selinux-6.5-20.el9cp.noarch nfs-ganesha-6.5-20.el9cp.x86_64 nfs-ganesha-ceph-6.5-20.el9cp.x86_64 nfs-ganesha-rados-grace-6.5-20.el9cp.x86_64 nfs-ganesha-rados-urls-6.5-20.el9cp.x86_64 nfs-ganesha-rgw-6.5-20.el9cp.x86_64 How reproducible: =============== Everytime Steps to Reproduce: =========== 1.Create NFS cluster with Encryption enabled 2. Create 2 NFS exports with --kmip_key_id set on creation # ceph fs subvolume getpath cephfs ganesha1 --group_name ganeshagroup /volumes/ganeshagroup/ganesha1/5a6b5e1b-4c86-4ec7-abbe-233d60bb3a98 # ceph fs subvolume getpath cephfs ganesha2 --group_name ganeshagroup /volumes/ganeshagroup/ganesha2/dac97290-00db-4934-b481-d846f38df7be # ceph nfs export create cephfs nfsganesha /ganesha1 cephfs --path=/volumes/ganeshagroup/ganesha1/5a6b5e1b-4c86-4ec7-abbe-233d60bb3a98 --kmip_key_id KEY-478ce1f-8c3310b3-0877-4893-b51e-527ba92c1d7d { "bind": "/ganesha1", "cluster": "nfsganesha", "fs": "cephfs", "mode": "RW", "path": "/volumes/ganeshagroup/ganesha1/5a6b5e1b-4c86-4ec7-abbe-233d60bb3a98" } # ceph nfs export create cephfs nfsganesha /ganesha2 cephfs --path=/volumes/ganeshagroup/ganesha2/dac97290-00db-4934-b481-d846f38df7be --kmip_key_id KEY-478ce1f-dbdf8785-9ff5-43e9-a81f-4090308e67ae { "bind": "/ganesha2", "cluster": "nfsganesha", "fs": "cephfs", "mode": "RW", "path": "/volumes/ganeshagroup/ganesha2/dac97290-00db-4934-b481-d846f38df7be" } 3.Restart the NFS service Actual results: ============= Ganesha is crashing on restart Expected results: ============= Ganesha should not crash Additional info: ============= Ganesha.log ----- nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] ceph_fsal_lookup :FSAL :F_DBG :Lookup .. Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] kmip_root_cb_func :FSAL :INFO :keyset success: kmip_key_id = KEY-478ce1f-8c3310b3-0877-4893-b51e-527ba92c1d7d, export = 1 Jun 24 20:19:48 ceph-hotfix-nfs-test-w3yafi-node2 ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv[2280688]: 24/06/2025 20:19:48 : epoch 685b085c : ceph-hotfix-nfs-test-w3yafi-node2 : ganesha.nfsd-2[main] ceph_fsal_lookup :FSAL :F_DBG :Lookup .. Jun 24 20:19:49 ceph-hotfix-nfs-test-w3yafi-node2 systemd-coredump[2280762]: Process 2280692 (ganesha.nfsd) of user 0 dumped core. Stack trace of thread 2: #0 0x00007f30b2b4028e n/a (/usr/lib64/libc.so.6 + 0x9a28e) ELF object binary architecture: AMD x86-64 Jun 24 20:19:49 ceph-hotfix-nfs-test-w3yafi-node2 podman[2280767]: 2025-06-24 20:19:49.774553703 +0000 UTC m=+0.030474121 container died a93e95f53bdec9512368b74336c50b9ec9059383190b2f1b460945815c184dd2 (image=cp.stg.icr.io/cp/ibm-ceph/ceph-8-rhel9@sha256:15e81a41f0c70565818a9c78cb79edf821606255f0ff42b298588586407f836f, name=ceph-c18432fc-4e36-11f0-b8b7-fa163ecefb6c-nfs-nfsganesha-0-0-ceph-hotfix-nfs-test-w3yafi-node2-tmusrv, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ibm-ceph/images/8-155.1.TEST.8.1z1, vcs-ref=3ece9b835558c52d067ecdc55b3bd164410a357f, release=155.1.TEST.8.1z1, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=IBM Storage Ceph 8, io.k8s.display-name=IBM Storage Ceph 8, description=IBM Storage Ceph 8, io.openshift.tags=ibm ceph, com.redhat.license_terms=https://www.redhat.com/agreements, CEPH_POINT_RELEASE=, architecture=x86_64, summary=Provides the latest IBM Storage Ceph 8 in a fully featured and supported base image., distribution-scope=public, vendor=Red Hat, Inc., version=8, RELEASE=main, io.openshift.expose-services=, GIT_COMMIT=eadbe5f6c4471e17c1721f9f08dde7964a4f491b, ceph=True, build-date=2025-06-20T21:01:18, GIT_CLEAN=True, GIT_BRANCH=main, vcs-type=git, maintainer=Guillaume Abrioux <gabrioux>, com.redhat.component=ibm-ceph-container, io.buildah.version=1.33.12, name=ibm-ceph)