Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2406405

Summary: [NFS-Ganesha] Ganesha crashes when the ingress mode is configured as haproxy-protocol.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Francesco Pantano <fpantano>
Component: NFS-GaneshaAssignee: Sachin Punadikar <spunadik>
NFS-Ganesha sub component: Ceph QA Contact: Manimaran M <manim>
Status: ASSIGNED --- Docs Contact:
Severity: high    
Priority: unspecified CC: akane, cephqe-warriors, ffilz, fpantano, gfidente, gouthamr, jcaratza, kkeithle, ltoscano, manim, mobisht, msaini, ngangadh, rpollack, spunadik, tserlin, vdas, vhariria
Version: 9.0   
Target Milestone: ---   
Target Release: 9.0z2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2338406 Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2338406    
Bug Blocks:    

Description Francesco Pantano 2025-10-26 15:27:10 UTC 
+++ This bug was initially created as a clone of Bug #2338406 +++

While this bug is created as a clone of #2338406, it's still not confirmed that is exactly the same issue.

After Ganesha is deployed with the regular cephadm command [1]:

```
- name: Create NFS Ganesha Cluster
  ansible.builtin.command: >
    {{ cifmw_cephadm_ceph_cli }} nfs cluster create {{ cifmw_cephadm_cephfs_name }}
    --ingress --virtual-ip={{ cifmw_cephadm_nfs_vip }}
    --ingress-mode=haproxy-protocol '--placement={{ placement }}'
  changed_when: false
  become: true
```
the instances serially start failing:

```
nfs.cephfs.0.0.ceph-b7q9enq2-1.pmrmst          ceph-b7q9enq2-1  *:12049               running (12m)     4m ago  22m    25.5M        -  7.0               e3263731880f  7857e4070ee7
nfs.cephfs.1.0.ceph-b7q9enq2-2.ngporp          ceph-b7q9enq2-2  *:12049               running (15m)    28s ago  22m    27.4M        -  7.0               e3263731880f  a5ea31cd8a05
nfs.cephfs.2.0.ceph-b7q9enq2-0.dobczm          ceph-b7q9enq2-0  *:12049               error             4m ago  22m        -        -  <unknown>         <unknown>     <unknown>
 
```

and the following log is observed:

```
[zuul@ceph-b7q9enq2-1 ~]$ sudo podman logs -f 7857e4070ee7                                                                                                                                                                17:13:16 [441/1931]
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 7.0
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :domainname in NFSv4 config section will soon be deprecated, define it under DIRECTORY_SERVICES section
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_user_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped users
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_group_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped groups
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] main :MAIN :WARN :Failed to set PR_SET_IO_FLUSHER due to EPERM, ignoring...
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] main :MAIN :EVENT :Using Bind_Addr as Monitoring_Addr
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576.
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] pwnam_wrappers__set_implementation :ID MAPPER :EVENT :NSSwitch pwnam implementation was loaded.
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Connection Manager initialized.
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs4_recovery_init :CLIENT ID :EVENT :Recovery Backend Init for rados_cluster
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] set_nodeid :CLIENT ID :EVENT :Nodeid : node0
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_init :CLIENT ID :EVENT :rados-cluster recovery backend initialization complete
24/10/2025 15:03:09 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:15 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:15 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:15 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:19 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:19 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:19 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:20 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:24 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:24 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:24 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:25 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:29 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:29 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:29 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:30 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:34 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:34 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:34 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:35 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:39 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:39 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:39 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:40 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:44 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:44 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:44 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:45 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:50 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:50 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:50 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:50 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:03:54 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:03:54 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:03:54 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:03:55 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:04:00 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:04:00 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:04:00 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:04:00 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:04:05 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:04:05 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:04:05 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:04:05 : epoch 68fb952d : ceph-b7q9enq2-1 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
....
```

```
[zuul@ceph-b7q9enq2-2 ~]$ sudo podman logs -f a5ea31cd8a05
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 7.0
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :domainname in NFSv4 config section will soon be deprecated, define it under DIRECTORY_SERVICES section
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_user_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped users
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_group_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped groups
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] main :MAIN :WARN :Failed to set PR_SET_IO_FLUSHER due to EPERM, ignoring...
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] main :MAIN :EVENT :Using Bind_Addr as Monitoring_Addr
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576.
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] pwnam_wrappers__set_implementation :ID MAPPER :EVENT :NSSwitch pwnam implementation was loaded.
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Connection Manager initialized.
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs4_recovery_init :CLIENT ID :EVENT :Recovery Backend Init for rados_cluster
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] set_nodeid :CLIENT ID :EVENT :Nodeid : node1
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_init :CLIENT ID :EVENT :rados-cluster recovery backend initialization complete
24/10/2025 14:59:32 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 14:59:38 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 14:59:38 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 14:59:38 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 14:59:42 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 14:59:42 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 14:59:42 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 14:59:43 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 14:59:47 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 14:59:47 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 14:59:47 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 14:59:48 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 14:59:53 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 14:59:53 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 14:59:53 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 14:59:53 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 14:59:58 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 14:59:58 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 14:59:58 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 14:59:58 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:00:03 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:00:03 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:00:03 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:00:03 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:00:08 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:00:08 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:00:08 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:00:08 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:00:12 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:00:12 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:00:12 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:00:13 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:00:17 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:00:17 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:00:17 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:00:18 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:00:22 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:00:22 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
24/10/2025 15:00:22 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
24/10/2025 15:00:23 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
24/10/2025 15:00:27 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
24/10/2025 15:00:27 : epoch 68fb9454 : ceph-b7q9enq2-2 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend

```

[1] https://github.com/openstack-k8s-operators/ci-framework/blob/main/roles/cifmw_cephadm/tasks/cephnfs.yml