Bug 2344746

Summary: [NFS-Ganesha] NFS ganesha deployment is failing with NFS v6.5-1.5 "Error while parsing RadosKV
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: NFS-GaneshaAssignee: Sachin Punadikar <spunadik>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.0CC: cephqe-warriors, kkeithle, spunadik, tserlin
Target Milestone: ---   
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfs-ganesha-6.5-9.el9cp Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-06-26 12:25:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2025-02-10 20:02:47 UTC
Description of problem:
============
Deploy NFS cluster with version v6.5-1.5. Ganesha process fails to come up

[ceph: root@argo016 /]# ceph nfs cluster create nfsganesha '1 argo018 argo019'

[ceph: root@argo016 /]# ceph nfs cluster info nfsganesha
{
  "nfsganesha": {
    "backend": [
      {
        "hostname": "argo018",
        "ip": "10.8.128.218",
        "port": 2049
      }
    ],
    "virtual_ip": null
  }
}

# ceph orch ps | grep nfs
nfs.nfsganesha.0.0.argo018.oumrdi  argo018  *:2049            unknown           7m ago   7m        -        -  <unknown>                                  <unknown>     <unknown>

ganesha.log
===========

Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 6.5
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :domainname in NFSv4 config section will soon be deprecated, define it under DIRECTORY_SERVICES section
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_user_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped users
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_group_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped groups
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] rados_load_config_from_parse :NFS STARTUP :CRIT :Error while parsing RadosKV specific configuration
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] main :NFS STARTUP :CRIT :Error setting parameters from configuration file.
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:20): Expected a number, got a double quoted string
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:18): 1 errors while processing parameters for RADOS_KV
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:18): Errors processing block (RADOS_KV)
Feb 10 19:50:24 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:24 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:30): 1 (invalid param value) errors found block RADOS_KV
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] main :NFS STARTUP :FATAL :Fatal errors.  Server exiting...
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libganesha_nfsd.so.6.5(+0x89671) [0x7efd6fd09671]
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libganesha_nfsd.so.6.5(+0x89aed) [0x7efd6fd09aed]
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libganesha_nfsd.so.6.5(DisplayLogComponentLevel+0x8b) [0x7efd6fd09edb]
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/usr/bin/ganesha.nfsd(main+0x71a) [0x5575ccfe8eaa]
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libc.so.6(+0x295d0) [0x7efd6fa815d0]
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libc.so.6(__libc_start_main+0x80) [0x7efd6fa81680]
Feb 10 19:50:25 argo018 ceph-ca667ce4-8c4a-11ef-ab13-ac1f6b0a1844-nfs-nfsganesha-0-0-argo018-oumrdi[487423]: 10/02/2025 19:50:25 : epoch 67aa5880 : argo018 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/usr/bin/ganesha.nfsd(_start+0x25) [0x5575ccfe9635]


Version-Release number of selected component (if applicable):
=====================

# rpm -qa | grep nfs
libnfsidmap-2.5.4-27.el9.x86_64
nfs-utils-2.5.4-27.el9.x86_64
nfs-ganesha-selinux-6.5-1.5.el9cp.noarch
nfs-ganesha-6.5-1.5.el9cp.x86_64
nfs-ganesha-ceph-6.5-1.5.el9cp.x86_64
nfs-ganesha-rados-grace-6.5-1.5.el9cp.x86_64
nfs-ganesha-rados-urls-6.5-1.5.el9cp.x86_64
nfs-ganesha-rgw-6.5-1.5.el9cp.x86_64
nfs-ganesha-utils-6.5-1.5.el9cp.x86_64



How reproducible:
==========
1/1


Steps to Reproduce:
===========
1. Deploy NFS Ganesha cluster

Actual results:
=========
Ganesha server failed to comeup. NFS Service is in failed state.


Expected results:
=======
Ganesha should be up and running


Additional info:

Comment 8 errata-xmlrpc 2025-06-26 12:25:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775