Bug 2309702

Summary: [Squid][NFS-Ganesha] NFS deployment is failing with latest 8.0 build , nfs-ganesha-6.0-3
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Kaleb KEITHLEY <kkeithle>
Component: NFS-GaneshaAssignee: Sachin Punadikar <spunadik>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0CC: akraj, cephqe-warriors, kkeithle, mbenjamin, tserlin, vdas
Target Milestone: ---Keywords: Automation, Regression, TestBlocker
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfs-ganesha-6.0-4.el9cp; rhceph-container-8-73 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-11-25 09:08:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2308414    

Description Kaleb KEITHLEY 2024-09-04 12:28:31 UTC 
This bug was initially created as a copy of Bug #2308414



NFS deployment is still failing with the fix

# ceph --version
ceph version 19.1.0-70.el9cp (00c94992708447e9731a83c63e7c826c0f1d8064) squid (rc)

# rpm -qa | grep nfs
libnfsidmap-2.5.4-26.el9_4.x86_64
nfs-utils-2.5.4-26.el9_4.x86_64
nfs-ganesha-selinux-6.0-3.el9cp.noarch
nfs-ganesha-6.0-3.el9cp.x86_64
nfs-ganesha-rgw-6.0-3.el9cp.x86_64
nfs-ganesha-ceph-6.0-3.el9cp.x86_64
nfs-ganesha-rados-grace-6.0-3.el9cp.x86_64
nfs-ganesha-rados-urls-6.0-3.el9cp.x86_64

1. Create NFS Ganesha cluster
# ceph nfs cluster create nfsganesha "ceph-nfs-test-v97d4j-node1-installer ceph-nfs-test-v97d4j-node2"

[ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph nfs cluster info nfsganesha
{
  "nfsganesha": {
    "backend": [
      {
        "hostname": "ceph-nfs-test-v97d4j-node1-installer",
        "ip": "10.0.64.70",
        "port": 2049
      },
      {
        "hostname": "ceph-nfs-test-v97d4j-node2",
        "ip": "10.0.66.230",
        "port": 2049
      }
    ],
    "virtual_ip": null
  }
}

2. Create export


[ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph nfs export create cephfs nfsganesha /ganesha1 cephfs --path=/
{
  "bind": "/ganesha1",
  "cluster": "nfsganesha",
  "fs": "cephfs",
  "mode": "RW",
  "path": "/"
}


[ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph nfs export info nfsganesha /ganesha1
{
  "access_type": "RW",
  "clients": [],
  "cluster_id": "nfsganesha",
  "export_id": 1,
  "fsal": {
    "cmount_path": "/",
    "fs_name": "cephfs",
    "name": "CEPH",
    "user_id": "nfs.nfsganesha.cephfs.2c1043d4"
  },
  "path": "/",
  "protocols": [
    3,
    4
  ],
  "pseudo": "/ganesha1",
  "security_label": true,
  "squash": "none",
  "transports": [
    "TCP"
  ]
}

3. Mount the export on client 

[root@ceph-nfs-test-v97d4j-node7 mnt]# mount -t nfs -o vers=4 10.0.64.70:/ganesha1 /mnt/ganesha/
mount.nfs: Connection refused

4. Ceph status

[ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph -s
  cluster:
    id:     0086c142-6a79-11ef-a4a1-fa163e50c07f
    health: HEALTH_WARN
            1 failed cephadm daemon(s)

  services:
    mon: 3 daemons, quorum ceph-nfs-test-v97d4j-node1-installer,ceph-nfs-test-v97d4j-node3,ceph-nfs-test-v97d4j-node2 (age 40m)
    mgr: ceph-nfs-test-v97d4j-node1-installer.mrrntp(active, since 41m), standbys: ceph-nfs-test-v97d4j-node3.xpdnwc
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 38m), 18 in (since 39m)
    rgw: 2 daemons active (2 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 274 objects, 468 KiB
    usage:   1.2 GiB used, 269 GiB / 270 GiB avail
    pgs:     721 active+clean

  io:
    client:   767 B/s rd, 0 op/s rd, 0 op/s wr


[ceph: root@ceph-nfs-test-v97d4j-node1-installer /]# ceph health detail
HEALTH_WARN 1 failed cephadm daemon(s)
[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
    daemon nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc on ceph-nfs-test-v97d4j-node1-installer is in error state
[ceph: root@ceph-nfs-test-v97d4j-node1-installer /]#


ceph orch ps | grep nfs
nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc  ceph-nfs-test-v97d4j-node1-installer  *:2049            error            71s ago  29m        -        -  <unknown>        <unknown>     <unknown>
nfs.nfsganesha.1.0.ceph-nfs-test-v97d4j-node2.amyedt            ceph-nfs-test-v97d4j-node2            *:2049            running (9m)     72s ago  29m    21.7M        -  6.0              ca14251cc89e  0cd87565447d



Observation -
=============

Ganesha service failed to start on the node and as a result mounting of export failed.



Ganesha.log
============

Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] rados_cluster_end_grace :CLIENT ID :EVENT :Failed to remove rec-0000000000000013:nfs.nfsganesha.0: -2
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:24): Unknown block (RADOS_URLS)
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:29): Unknown block (RGW)
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory)
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc[78218]: 04/09/2024 05:24:00 : epoch 66d7eed8 : ceph-nfs-test-v97d4j-node1-installer : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer podman[84982]: 2024-09-04 01:24:00.572190635 -0400 EDT m=+0.025885387 container died ef1503742addf568276f97f913a8708b114aa6a4d311bf967a24cb64222827b1 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2d153b5d847e1e44ae72a059e1078e191c8a78f72fd469e907f09ad77e235de8, name=ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc, vcs-type=git, io.openshift.tags=rhceph ceph, io.openshift.expose-services=, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, name=rhceph, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., io.k8s.description=Red Hat Ceph Storage 8, release=48, version=8, RELEASE=main, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, com.redhat.license_terms=https://www.redhat.com/agreements, ceph=True, maintainer=Guillaume Abrioux <gabrioux>, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-48, vendor=Red Hat, Inc., GIT_BRANCH=main, CEPH_POINT_RELEASE=, build-date=2024-09-03T22:57:55, description=Red Hat Ceph Storage 8, io.buildah.version=1.29.0, com.redhat.component=rhceph-container, vcs-ref=f0f2707c29c8affe98c484af48cf2d3b5459146f, architecture=x86_64, distribution-scope=public, GIT_CLEAN=True)
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer podman[84982]: 2024-09-04 01:24:00.591276737 -0400 EDT m=+0.044971482 container remove ef1503742addf568276f97f913a8708b114aa6a4d311bf967a24cb64222827b1 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2d153b5d847e1e44ae72a059e1078e191c8a78f72fd469e907f09ad77e235de8, name=ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f-nfs-nfsganesha-0-0-ceph-nfs-test-v97d4j-node1-installer-mwfiqc, build-date=2024-09-03T22:57:55, io.openshift.tags=rhceph ceph, maintainer=Guillaume Abrioux <gabrioux>, io.buildah.version=1.29.0, com.redhat.license_terms=https://www.redhat.com/agreements, io.openshift.expose-services=, vendor=Red Hat, Inc., ceph=True, CEPH_POINT_RELEASE=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-48, vcs-type=git, io.k8s.description=Red Hat Ceph Storage 8, GIT_CLEAN=True, vcs-ref=f0f2707c29c8affe98c484af48cf2d3b5459146f, architecture=x86_64, release=48, distribution-scope=public, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., com.redhat.component=rhceph-container, description=Red Hat Ceph Storage 8, GIT_BRANCH=main, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, GIT_REPO=https://github.com/ceph/ceph-container.git, name=rhceph, version=8, RELEASE=main)
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Main process exited, code=exited, status=139/n/a
Sep 04 01:24:00 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Failed with result 'exit-code'.
Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Scheduled restart job, restart counter is at 5.
Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: Stopped Ceph nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc for 0086c142-6a79-11ef-a4a1-fa163e50c07f.
Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Start request repeated too quickly.
Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: ceph-0086c142-6a79-11ef-a4a1-fa163e50c07f.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc.service: Failed with result 'exit-code'.
Sep 04 01:24:10 ceph-nfs-test-v97d4j-node1-installer systemd[1]: Failed to start Ceph nfs.nfsganesha.0.0.ceph-nfs-test-v97d4j-node1-installer.mwfiqc for 0086c142-6a79-11ef-a4a1-fa163e50c07f.
[root@ceph-nfs-test-v97d4j-node1-installer ~]#
Comment 9 errata-xmlrpc 2024-11-25 09:08:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:10216