Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use Jira Cloud for all bug tracking management.

Bug 2375834

Summary:	[8.1z][NFS-Ganesha][BYOK] Git clone hangs indefinitely on BYOK-enabled NFS exports
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Manisha Saini <msaini>
Component:	NFS-Ganesha	Assignee:	Marcus Watts <mwatts>
Status:	VERIFIED ---	QA Contact:	Manisha Saini <msaini>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	8.1	CC:	aravindr, bkunal, cephqe-warriors, deepatil, ffilz, hyelloji, kkeithle, mamohan, mbenjamin, spunadik, tserlin, vshankar
Target Milestone:	---	Keywords:	External
Target Release:	8.1z2	Flags:	deepatil: needinfo-
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	nfs-ganesha-6.5-24.el9cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2379790 (view as bug list)		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2379790
Bug Blocks:

Description Manisha Saini 2025-07-01 23:42:23 UTC

Description of problem:
======================

When NFS exports are configured with BYOK  encryption enabled via KMIP, running git clone on the mounted directory results in the command hanging indefinitely.


Version-Release number of selected component (if applicable):
===========================
# rpm -qa | grep nfs
libnfsidmap-2.5.4-34.el9.x86_64
nfs-utils-2.5.4-34.el9.x86_64
nfs-ganesha-selinux-6.5-23.el9cp.noarch
nfs-ganesha-6.5-23.el9cp.x86_64
nfs-ganesha-ceph-6.5-23.el9cp.x86_64
nfs-ganesha-rados-grace-6.5-23.el9cp.x86_64
nfs-ganesha-rados-urls-6.5-23.el9cp.x86_64
nfs-ganesha-rgw-6.5-23.el9cp.x86_64


How reproducible:
===============
2/2


Steps to Reproduce:
=================

1. Configure a CephFS export via NFS-Ganesha with BYOK (KMIP key, cert, and CA) enabled.

# ceph nfs cluster create nfsganesha "cali016" -i /var/lib/ceph/gklm.yaml

# ceph fs subvolume create cephfs ganesha1 --group_name ganeshagroup --enctag KEY-478ce1f-7373659b-f724-49a7-b936-662ce3dd1129

# ceph fs subvolume getpath cephfs ganesha1 --group_name ganeshagroup
/volumes/ganeshagroup/ganesha1/fa20d714-d99f-4147-b451-2e3d6ba97691

# ceph nfs export create cephfs nfsganesha /ganesha1 cephfs --path /volumes/ganeshagroup/ganesha1/fa20d714-d99f-4147-b451-2e3d6ba97691 --kmip_key_id  KEY-478ce1f-7373659b-f724-49a7-b936-662ce3dd1129
{
  "bind": "/ganesha1",
  "cluster": "nfsganesha",
  "fs": "cephfs",
  "mode": "RW",
  "path": "/volumes/ganeshagroup/ganesha1/fa20d714-d99f-4147-b451-2e3d6ba97691"
}

# ceph nfs cluster info nfsganesha
{
  "nfsganesha": {
    "backend": [
      {
        "hostname": "cali016",
        "ip": "10.8.130.16",
        "port": 2049
      }
    ],
    "virtual_ip": null
  }
}

2. Mount the export on a client using via vers 4 and create file and dirs. Perform git clone on NFS mount point

[root@ceph-doremon-nfs-w6q886-node4 mnt]# mount -t nfs 10.8.130.16:/ganesha1 /mnt/ganesha/
[root@ceph-doremon-nfs-w6q886-node4 mnt]# cd /mnt/ganesha/
[root@ceph-doremon-nfs-w6q886-node4 ganesha]# ls
[root@ceph-doremon-nfs-w6q886-node4 ganesha]# mkdir dir1
[root@ceph-doremon-nfs-w6q886-node4 ganesha]# touch f1
[root@ceph-doremon-nfs-w6q886-node4 ganesha]# ls
dir1  f1
[root@ceph-doremon-nfs-w6q886-node4 ganesha]# git clone git://git.linux-nfs.org/projects/bfields/pynfs.git
Cloning into 'pynfs'...


From ceph-fuse mount
---------

[root@ceph-doremon-nfs-w6q886-node6 mnt]# ceph-fuse -n client.admin /mnt/cephfs/ -r /volumes/ganeshagroup/ganesha1/fa20d714-d99f-4147-b451-2e3d6ba97691 --client_fs=cephfs
2025-07-01T23:09:59.786+0000 7f51bf22c580 -1 init, newargv = 0x56054988e680 newargc=15
ceph-fuse[92137]: starting ceph client
ceph-fuse[92137]: starting fuse

[root@ceph-doremon-nfs-w6q886-node6 mnt]# cd /mnt/cephfs/


[root@ceph-doremon-nfs-w6q886-node6 cephfs]# ls
1crZ2y20Z4hC7nD8WuxbQdOWkcZvpUzQ71KsrZ52eek  8QS0mSlZHHXOoxNBo5GD6Vce80GnM3R36Id5Db+O5tk  OQgPTvhnXOloDk3OJxqPTfCVitdL5jxRiVNUyvzvZsg


Mounted the same export on another client and retried the git clone operation
-----------------------------
[root@ceph-doremon-nfs-w6q886-node5 mnt]# mount -t nfs 10.8.130.16:/ganesha1 /mnt/ganesha/

[root@ceph-doremon-nfs-w6q886-node5 mnt]#

[root@ceph-doremon-nfs-w6q886-node5 mnt]# cd /mnt/ganesha/

[root@ceph-doremon-nfs-w6q886-node5 ganesha]# ls
dir1  f1  pynfs

[root@ceph-doremon-nfs-w6q886-node5 ganesha]# cd pynfs/

[root@ceph-doremon-nfs-w6q886-node5 pynfs]# ls.   =================> This was empty directory created as part of previous git clone command

[root@ceph-doremon-nfs-w6q886-node5 pynfs]#

[root@ceph-doremon-nfs-w6q886-node5 pynfs]# cd ..

[root@ceph-doremon-nfs-w6q886-node5 ganesha]# ls
dir1  f1  pynfs

[root@ceph-doremon-nfs-w6q886-node5 ganesha]# cd dir1/

[root@ceph-doremon-nfs-w6q886-node5 dir1]# ls


[root@ceph-doremon-nfs-w6q886-node5 dir1]# git clone git://git.linux-nfs.org/projects/bfields/pynfs.git
Cloning into 'pynfs'...




^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
[root@ceph-doremon-nfs-w6q886-node5 dir1]# ls




Actual results:
================
The git clone hangs indefinitely, with no visible progress or error. Strace shows blocking I/O calls.
Created fresh export without BYOK and tried mounting , that too hangs now


Expected results:
=================
The git clone should complete successfully and the repository should be cloned onto the mounted export.

Additional info:
===============

[ceph: root@cali013 /]# ceph orch ps | grep nfsganesha
nfs.nfsganesha.0.0.cali016.mzzfst  cali016  *:2049            running (32m)     2m ago  32m     122M        -  6.5               b8860365707a  39daf7bd5133

[ceph: root@cali013 /]# ceph orch ls
NAME                       PORTS        RUNNING  REFRESHED  AGE  PLACEMENT
alertmanager               ?:9093,9094      1/1  5m ago     2w   count:1
ceph-exporter                               5/5  5m ago     2w   *
crash                                       5/5  5m ago     2w   *
grafana                    ?:3000           1/1  5m ago     2w   count:1
mds.cephfs                                  2/2  5m ago     2w   count:2
mgr                                         2/2  5m ago     2w   count:2
mon                                         5/5  5m ago     2w   count:5
nfs.nfsganesha             ?:2049           1/1  2m ago     33m  cali016
node-exporter              ?:9100           5/5  5m ago     2w   *
osd.all-available-devices                    27  5m ago     2w   *
prometheus                 ?:9095           1/1  5m ago     2w   count:1

[ceph: root@cali013 /]# ceph -s
  cluster:
    id:     d62df802-4759-11f0-b17e-b49691cee574
    health: HEALTH_OK

  services:
    mon: 5 daemons, quorum cali013,cali016,cali015,cali020,cali019 (age 55m)
    mgr: cali013.napmfg(active, since 40h), standbys: cali016.tjreee
    mds: 1/1 daemons up, 1 standby
    osd: 27 osds: 27 up (since 55m), 27 in (since 8d)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 1073 pgs
    objects: 106 objects, 197 MiB
    usage:   21 GiB used, 66 TiB / 67 TiB avail
    pgs:     1073 active+clean

  io:
    client:   85 B/s rd, 0 op/s rd, 0 op/s wr