This bug was initially created as a copy of Bug #2308414 NFS deployment is still failing with the fix # ceph --version ceph version 19.1.0-70.el9cp (00c94992708447e9731a83c63e7c826c0f1d8064) squid (rc) # rpm -qa | grep nfs libnfsidmap-2.5.4-26.el9_4.x86_64 nfs-utils-2.5.4-26.el9_4.x86_64 nfs-ganesha-selinux-6.0-3.el9cp.noarch nfs-ganesha-6.0-3.el9cp.x86_64 nfs-ganesha-rgw-6.0-3.el9cp.x86_64 nfs-ganesha-ceph-6.0-3.el9cp.x86_64 nfs-ganesha-rados-grace-6.0-3.el9cp.x86_64 nfs-ganesha-rados-urls-6.0-3.el9cp.x86_64 1. Create NFS Ganesha cluster # ceph nfs cluster create nfsganesha "ceph-nfs-test-v97d4j-node1-installer ceph-nfs-test-v97d4j-node2" [ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph nfs cluster info nfsganesha { "nfsganesha": { "backend": [ { "hostname": "ceph-nfs-test-v97d4j-node1-installer", "ip": "10.0.64.70", "port": 2049 }, { "hostname": "ceph-nfs-test-v97d4j-node2", "ip": "10.0.66.230", "port": 2049 } ], "virtual_ip": null } } 2. Create export [ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph nfs export create cephfs nfsganesha /ganesha1 cephfs --path=/ { "bind": "/ganesha1", "cluster": "nfsganesha", "fs": "cephfs", "mode": "RW", "path": "/" } [ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph nfs export info nfsganesha /ganesha1 { "access_type": "RW", "clients": [], "cluster_id": "nfsganesha", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "cephfs", "name": "CEPH", "user_id": "nfs.nfsganesha.cephfs.2c1043d4" }, "path": "/", "protocols": [ 3, 4 ], "pseudo": "/ganesha1", "security_label": true, "squash": "none", "transports": [ "TCP" ] } 3. Mount the export on client [root@ceph-nfs-test-v97d4j-node7 mnt]# mount -t nfs -o vers=4 10.0.64.70:/ganesha1 /mnt/ganesha/ mount.nfs: Connection refused 4. Ceph status [ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph -s cluster: id: 0086c142-6a79-11ef-a4a1-fa163e50c07f health: HEALTH_WARN 1 failed cephadm daemon(s) services: mon: 3 daemons, quorum ceph-nfs-test-v97d4j-node1-installer,ceph-nfs-test-v97d4j-node3,ceph-nfs-test-v97d4j-node2 (age 40m) mgr: ceph-nfs-test-v97d4j-node1-installer.mrrntp(active, since 41m), standbys: ceph-nfs-test-v97d4j-node3.xpdnwc mds: 1/1 daemons up, 1 standby osd: 18 osds: 18 up (since 38m), 18 in (since 39m) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 9 pools, 721 pgs objects: 274 objects, 468 KiB usage: 1.2 GiB used, 269 GiB / 270 GiB avail pgs: 721 active+clean io: client: 767 B/s rd, 0 op/s rd, 0 op/s wr [ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc on ceph-nfs-test-v97d4j-node1-installer is in error state [ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph orch ps | grep nfs nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc ceph-nfs-test-v97d4j-node1-installer *:2049 error 71s ago 29m - - <unknown> <unknown> <unknown> nfs.nfsganesha.1.0.ceph-nfs-test-v97d4j-node2.amyedt ceph-nfs-test-v97d4j-node2 *:2049 running (9m) 72s ago 29m 21.7M - 6.0 ca14251cc89e 0cd87565447d Observation - ============= Ganesha service failed to start on the node and as a result mounting of export failed. Ganesha.log ============ Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] rados_cluster_end_grace :CLIENT ID :EVENT :Failed to remove rec-0000000000000013:nfs.nfsganesha.0: -2 Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:24): Unknown block (RADOS_URLS) Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:29): Unknown block (RGW) Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer podman[84982]: 2024-09-04 01:24:00.572190635 -0400 EDT m=+0.025885387 container died ef1503742addf568276f97f913a8708b114aa6a4d311bf967a24cb64222827b1 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2d153b5d847e1e44ae72a059e1078e191c8a78f72fd469e907f09ad77e235de8, name=ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc, vcs-type=git, io.openshift.tags=rhceph ceph, io.openshift.expose-services=, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, name=rhceph, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., io.k8s.description=Red Hat Ceph Storage 8, release=48, version=8, RELEASE=main, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, com.redhat.license_terms=https://www.redhat.com/agreements, ceph=True, maintainer=Guillaume Abrioux <gabrioux>, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-48, vendor=Red Hat, Inc., GIT_BRANCH=main, CEPH_POINT_RELEASE=, build-date=2024-09-03T22:57:55, description=Red Hat Ceph Storage 8, io.buildah.version=1.29.0, com.redhat.component=rhceph-container, vcs-ref=f0f2707c29c8affe98c484af48cf2d3b5459146f, architecture=x86_64, distribution-scope=public, GIT_CLEAN=True) Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer podman[84982]: 2024-09-04 01:24:00.591276737 -0400 EDT m=+0.044971482 container remove ef1503742addf568276f97f913a8708b114aa6a4d311bf967a24cb64222827b1 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2d153b5d847e1e44ae72a059e1078e191c8a78f72fd469e907f09ad77e235de8, name=ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc, build-date=2024-09-03T22:57:55, io.openshift.tags=rhceph ceph, maintainer=Guillaume Abrioux <gabrioux>, io.buildah.version=1.29.0, com.redhat.license_terms=https://www.redhat.com/agreements, io.openshift.expose-services=, vendor=Red Hat, Inc., ceph=True, CEPH_POINT_RELEASE=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-48, vcs-type=git, io.k8s.description=Red Hat Ceph Storage 8, GIT_CLEAN=True, vcs-ref=f0f2707c29c8affe98c484af48cf2d3b5459146f, architecture=x86_64, release=48, distribution-scope=public, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., com.redhat.component=rhceph-container, description=Red Hat Ceph Storage 8, GIT_BRANCH=main, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, GIT_REPO=https://github.com/ceph/ceph-container.git, name=rhceph, version=8, RELEASE=main) Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Main process exited, code=exited, status=139/n/a Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Failed with result 'exit-code'. Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Scheduled restart job, restart counter is at 5. Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: Stopped Ceph nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc for 0086c142-6a79-11ef-a4a1-fa163e50c07f. Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Start request repeated too quickly. Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Failed with result 'exit-code'. Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: Failed to start Ceph nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc for 0086c142-6a79-11ef-a4a1-fa163e50c07f. [root@ceph-nfs-test-v97d4j-node1-installer ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:10216