Bug 2413977
| Summary: | [NFS-Ganesha] [BYOK] NFS mount becomes inaccessible after restarting Ganesha when BYOK is enabled | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Manisha Saini <msaini> |
| Component: | NFS-Ganesha | Assignee: | Marcus Watts <mwatts> |
| NFS-Ganesha sub component: | Ceph | QA Contact: | |
| Status: | CLOSED UPSTREAM | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | cephqe-warriors, ffilz, hacharya, kkeithle, nchillar, ngangadh |
| Version: | 9.0 | Keywords: | Regression |
| Target Milestone: | --- | ||
| Target Release: | 9.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2026-03-04 09:57:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2385962 | ||
This product has been discontinued or is no longer tracked in Red Hat Bugzilla. |
Description of problem: ================= When the Ceph NFS service is deployed with BYOK enabled, the mounted NFS export works initially. However, after restarting the Ganesha service, the NFS mount becomes inaccessible on the client side, and subsequent I/O operations fail with a “Stale file handle” error. ------- 11/11/2025 06:38:05 : epoch 6912d9cc : ceph-manisaini-zc133g-node2 : ganesha.nfsd-2[main] kmip_root_cb_func :FSAL :CRIT :keyset failed: kmip_key_id = KEY-3d4a967-0cc98f21-7d9e-45b0-a0c8-040acead4317, export = 1, error = 17/17 (existing policy mismatch) ------- Version-Release number of selected component (if applicable): ================================ # rpm -qa | grep nfs libnfsidmap-2.8.2-3.el10.x86_64 nfs-utils-2.8.2-3.el10.x86_64 nfs-ganesha-selinux-7.0-0.6.5.el10cp.noarch nfs-ganesha-7.0-0.6.5.el10cp.x86_64 nfs-ganesha-ceph-7.0-0.6.5.el10cp.x86_64 nfs-ganesha-rados-grace-7.0-0.6.5.el10cp.x86_64 nfs-ganesha-rados-urls-7.0-0.6.5.el10cp.x86_64 nfs-ganesha-rgw-7.0-0.6.5.el10cp.x86_64 nfs-ganesha-utils-7.0-0.6.5.el10cp.x86_64 # ceph --version ceph version 20.1.0-78.el10cp (28fe4cdcd151a754a1cc69c1ee09d5d9657f2a4c) tentacle (rc) How reproducible: ============= 2/2 Steps to Reproduce: ================== 1. Deploy NFS cluster with BYOK enabled: [ceph: root@ceph-manisaini-zc133g-node1-installer /]# ceph orch apply -i /var/lib/ceph/deploy_cluster_byok.yaml Scheduled nfs.nfsganesha update... [ceph: root@ceph-manisaini-zc133g-node1-installer /]# ceph nfs cluster info nfsganesha { "nfsganesha": { "backend": [ { "hostname": "ceph-manisaini-zc133g-node2", "ip": "10.0.64.35", "port": 2049 } ], "virtual_ip": null } } 2.Create a CephFS subvolume and export with KMIP key ID: [ceph: root@ceph-manisaini-zc133g-node1-installer /]# ceph fs subvolume create cephfs ganesha1 --group_name ganeshagroup --namespace-isolated [ceph: root@ceph-manisaini-zc133g-node1-installer /]# ceph fs subvolume getpath cephfs ganesha1 --group_name ganeshagroup /volumes/ganeshagroup/ganesha1/028feef3-360d-4171-9e7d-53859ccf2471 [ceph: root@ceph-manisaini-zc133g-node1-installer /]# ceph nfs export create cephfs nfsganesha /ganesha1 cephfs --path /volumes/ganeshagroup/ganesha1/028feef3-360d-4171-9e7d-53859ccf2471 --kmip_key_id KEY-3d4a967-0cc98f21-7d9e-45b0-a0c8-040acead4317 { "bind": "/ganesha1", "cluster": "nfsganesha", "fs": "cephfs", "mode": "RW", "path": "/volumes/ganeshagroup/ganesha1/028feef3-360d-4171-9e7d-53859ccf2471" } ---- Ganesha.log ---- 11/11/2025 06:28:28 : epoch 6912d73b : ceph-manisaini-zc133g-node2 : ganesha.nfsd-2[sigmgr] reclaim_reset :FSAL :EVENT :start_reclaim failed: (-2) No such file or directory 11/11/2025 06:28:28 : epoch 6912d73b : ceph-manisaini-zc133g-node2 : ganesha.nfsd-2[sigmgr] kmip_root_cb_func :FSAL :EVENT :keyset success: kmip_key_id = KEY-3d4a967-0cc98f21-7d9e-45b0-a0c8-040acead4317, export = 1 3.Mount the export and perform I/O: [root@ceph-manisaini-zc133g-node6 mnt]# mount -t nfs 10.0.64.35:/ganesha1 /mnt/ganesha1/ [root@ceph-manisaini-zc133g-node6 mnt]# cd /mnt/ganesha1/ [root@ceph-manisaini-zc133g-node6 ganesha1]# ls [root@ceph-manisaini-zc133g-node6 ganesha1]# mkdir dir1 [root@ceph-manisaini-zc133g-node6 ganesha1]# touch f1 [root@ceph-manisaini-zc133g-node6 ganesha1]# dd if=/dev/urandom of=/mnt/ganesha1/file1 bs=1G count=1 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.34829 s, 129 MB/s 4. Restart Ganesha: [ceph: root@ceph-manisaini-zc133g-node1-installer /]# ceph orch restart nfs.nfsganesha Scheduled to restart nfs.nfsganesha.0.0.ceph-manisaini-zc133g-node2.zgozvg on host 'ceph-manisaini-zc133g-node2' [root@ceph-manisaini-zc133g-node2 ~]# pgrep ganesha 513963 ---- Ganesha.log ---- 11/11/2025 06:38:05 : epoch 6912d9cc : ceph-manisaini-zc133g-node2 : ganesha.nfsd-2[main] kmip_root_cb_func :FSAL :CRIT :keyset failed: kmip_key_id = KEY-3d4a967-0cc98f21-7d9e-45b0-a0c8-040acead4317, export = 1, error = 17/17 (existing policy mismatch) 5. Again write 1 file on the mount point [root@ceph-manisaini-zc133g-node6 ganesha1]# dd if=/dev/urandom of=/mnt/ganesha1/file2 bs=1G count=1 dd: failed to open '/mnt/ganesha1/file2': Stale file handle Actual results: ============== After restarting Ganesha, NFS export becomes inaccessible. Client operations fail with “Stale file handle”. Ganesha.log shows - FSAL :CRIT :keyset failed: kmip_key_id = KEY-3d4a967-0cc98f21-7d9e-45b0-a0c8-040acead4317, export = 1, error = 17/17 (existing policy mismatch) Expected results: ================ NFS export should remain accessible after Ganesha restart. Client mounts should continue to function without errors. Additional info: