Description of problem: ============= Updating the export config to change the "RW" permission to "RO" is failing with validation error in export block. ============ n] create_export :FSAL :CRIT :Unable to init Ceph handle. n] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph n] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/exportRO) to (/) n] export_commit_common :CONFIG :CRIT :fsal_export is NULL n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":2): 1 validation errors in block FSAL n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":2): Errors processing block (FSAL) n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":1): 1 validation errors in block EXPORT n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":1): Errors processing block (EXPORT) n] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:22): Unknown block (RADOS_URLS) n] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:27): Unknown block (RGW) n] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL ============= Version-Release number of selected component (if applicable): ========================= # rpm -qa | grep nfs libnfsidmap-2.5.4-18.el9.x86_64 nfs-utils-2.5.4-18.el9.x86_64 nfs-ganesha-selinux-5.6-3.el9cp.noarch nfs-ganesha-5.6-3.el9cp.x86_64 nfs-ganesha-rgw-5.6-3.el9cp.x86_64 nfs-ganesha-ceph-5.6-3.el9cp.x86_64 nfs-ganesha-rados-grace-5.6-3.el9cp.x86_64 nfs-ganesha-rados-urls-5.6-3.el9cp.x86_64 # ceph --version ceph version 18.2.0-113.el9cp (32cbda69435c7145d09eeaf5b5016e5d46370a5d) reef (stable) How reproducible: ========= 2/2 Steps to Reproduce: ============ 1.Create nfs ganesha cluster # ceph nfs cluster info { "cephfs-nfs": { "backend": [ { "hostname": "ceph-doremon-akb3kk-node2", "ip": "10.0.211.53", "port": 2049 } ], "virtual_ip": null } } 2.Create an export # ceph nfs export create cephfs cephfs-nfs /exportRO cephfs { "bind": "/exportRO", "cluster": "cephfs-nfs", "fs": "cephfs", "mode": "RW", "path": "/" } # ceph nfs export get cephfs-nfs /exportRO { "access_type": "RW", "clients": [], "cluster_id": "cephfs-nfs", "export_id": 2, "fsal": { "cmount_path": "/", "fs_name": "cephfs", "name": "CEPH", "user_id": "nfs.cephfs-nfs.cephfs" }, "path": "/", "protocols": [ 4 ], "pseudo": "/exportRO", "security_label": true, "squash": "none", "transports": [ "TCP" ] } 4. Mount the export on client. 5. Update the export with "RO" permission. [ceph: root@ceph-doremon-akb3kk-node1-installer /]# ceph nfs export get cephfs-nfs /exportRO > export.conf [ceph: root@ceph-doremon-akb3kk-node1-installer /]# sed -i 's/"access_type": "RW"/"access_type": "RO"/g' export.conf [ceph: root@ceph-doremon-akb3kk-node1-installer /]# ceph nfs export apply cephfs-nfs -i export.conf [ { "pseudo": "/exportRO", "state": "updated" } ] [ceph: root@ceph-doremon-akb3kk-node1-installer /]# ceph nfs export get cephfs-nfs /exportRO { "access_type": "RO", "clients": [], "cluster_id": "cephfs-nfs", "export_id": 2, "fsal": { "cmount_path": "/", "fs_name": "cephfs", "name": "CEPH", "user_id": "nfs.cephfs-nfs.cephfs" }, "path": "/", "protocols": [ 4 ], "pseudo": "/exportRO", "security_label": true, "squash": "none", "transports": [ "TCP" ] } 6. Remount the volume on client. Actual results: ============ Mount failed.Observed the validation error in ganesha.log Expected results: =========== Mount should succeed. Additional info: ============ ganesha.log ---- n] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 n] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. n] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. n] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. n] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 n] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend n] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) n] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE n] create_export :FSAL :CRIT :Unable to init Ceph handle. n] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph n] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/exportRO) to (/) n] export_commit_common :CONFIG :CRIT :fsal_export is NULL n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":2): 1 validation errors in block FSAL n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":2): Errors processing block (FSAL) n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":1): 1 validation errors in block EXPORT n] config_errs_to_log :CONFIG :CRIT :Config File ("rados://.nfs/cephfs-nfs/export-2":1): Errors processing block (EXPORT) n] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:22): Unknown block (RADOS_URLS) n] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:27): Unknown block (RGW) n] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL n] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_> n] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) n] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection n] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection n] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection n] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs n] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection n] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized n] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists n] find_keytab_entry :NFS CB :WARN :Configuration file does not specify default realm while getting default realm name n] gssd_refresh_krb5_machine_credential :NFS CB :CRIT :ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keyta> n] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:2) n] nfs_Start_threads :THREAD :EVENT :Starting delayed executor. n] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully n] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully n] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully n] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully n] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- n] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED n] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- s] gsh_dbus_thread :DBUS :CRIT :DBUS not initialized, service thread exiting s] gsh_dbus_thread :DBUS :EVENT :shutdown
@Manisha Saini can you get the raw ganesha config blob? Here's how I typically do it: Install `rados` binary or use `cephadm shell`, then: # list all namespaces/objects rados --pool .nfs --all ls # display the content of one object rados --pool .nfs -N <cluster_id> get export-<export_id> /dev/stdout (example) rados --pool .nfs -N cephfs-nfs get export-2 /dev/stdout
Following the steps above I encountered _an_ error. After using the 'apply' command to change an export to RO, my config blobs looked as follows: [ceph: root@ceph0 /]# rados --pool .nfs -N nfs1 get export-2 /tmp/e2 [ceph: root@ceph0 /]# rados --pool .nfs -N nfs1 get export-1 /tmp/e1 [ceph: root@ceph0 /]# cat /tmp/e1 /tmp/e2 EXPORT { FSAL { name = "CEPH"; user_id = "nfs.nfs1.cephfs"; filesystem = "cephfs"; secret_access_key = "AQCOREllIsGGGBAAeeW08tMygJGufbxwtpTppg=="; cmount_path = "/"; } export_id = 1; path = "/"; pseudo = "/fs1"; access_type = "RW"; squash = "none"; attr_expiration_time = 0; security_label = true; protocols = 4; transports = "TCP"; } EXPORT { FSAL { name = "CEPH"; user_id = "nfs.nfs1.cephfs"; filesystem = "cephfs"; cmount_path = "/"; } export_id = 2; path = "/b"; pseudo = "/b"; access_type = "RO"; squash = "none"; attr_expiration_time = 0; security_label = true; protocols = 4; transports = "TCP"; } Note that export 2 lacks a secret_access_key parameter. IMPORTANT: my build did not have the following patch https://github.com/ceph/ceph/pull/54277/commits/80f2e3ce1fee21443b7e0ea00cc0dcc835e61789 This patch is necessary for the dashboard and *may be* related. Does the build used in this test scenario have this patch or the equivalent?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:7780