Bug 2369746
| Summary: | [8.0z5] [NFS-Ganesha]NFS ganesha cluster creation is failing on the upgraded cluster from 7.x to 8.0z5 | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Manisha Saini <msaini> | |
| Component: | NFS-Ganesha | Assignee: | Sachin Punadikar <spunadik> | |
| Status: | CLOSED ERRATA | QA Contact: | Manisha Saini <msaini> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 8.0 | CC: | cephqe-warriors, hacharya, kkeithle, tserlin | |
| Target Milestone: | --- | Keywords: | Automation, Regression | |
| Target Release: | 8.0z5 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | nfs-ganesha-6.5-12.5.el9cp; rhceph-container-8-480 | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2369820 (view as bug list) | Environment: | ||
| Last Closed: | 2025-06-09 14:17:05 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2369412, 2369465, 2369820 | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 8.0 bug fix updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2025:8694 |
Description of problem: ======== On the upgraded cluster from 7.x to 8.0z5, NFS Ganesha cluster creation is failing. [ceph: root@ceph-nfsupgradetest-fu6x8g-node1-installer /]# ceph nfs cluster create nfsganesha "ceph-nfsupgradetest-fu6x8g-node2" [ceph: root@ceph-nfsupgradetest-fu6x8g-node1-installer /]# ceph nfs cluster info nfsganesha { "nfsganesha": { "backend": [ { "hostname": "ceph-nfsupgradetest-fu6x8g-node2", "ip": "10.0.64.202", "port": 2049 } ], "virtual_ip": null } } [ceph: root@ceph-nfsupgradetest-fu6x8g-node1-installer /]# ceph orch ls | grep nfs nfs.nfsganesha ?:2049 0/1 56s ago 60s ceph-nfsupgradetest-fu6x8g-node2 [ceph: root@ceph-nfsupgradetest-fu6x8g-node1-installer /]# ceph orch ps | grep nfsganesha nfs.nfsganesha.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz ceph-nfsupgradetest-fu6x8g-node2 *:2049 unknown 2m ago 2m - - <unknown> <unknown> <unknown> [ceph: root@ceph-nfsupgradetest-fu6x8g-node1-installer /]# Version-Release number of selected component (if applicable): ========== # rpm -qa | grep nfs libnfsidmap-2.5.4-34.el9.x86_64 nfs-utils-2.5.4-34.el9.x86_64 nfs-ganesha-selinux-6.5-12.3.el9cp.noarch nfs-ganesha-6.5-12.3.el9cp.x86_64 nfs-ganesha-rgw-6.5-12.3.el9cp.x86_64 nfs-ganesha-ceph-6.5-12.3.el9cp.x86_64 nfs-ganesha-rados-grace-6.5-12.3.el9cp.x86_64 nfs-ganesha-rados-urls-6.5-12.3.el9cp.x86_64 nfs-ganesha-utils-6.5-12.3.el9cp.x86_64 # ceph --version ceph version 19.2.0-139.el9cp (d625bfed07c649268e36061ca769a5ffc77e797a) squid (stable) How reproducible: ======= 2/2 Steps to Reproduce: ========= 1. Create NFS Ganesha cluster on 7.x 2. Create 3 NFS exports and mount it over clients 3. Run some IO's and lookups 4. Upgrade the cluster to 8.0z5 builds while IO's are running 5. Delete the existing NFS Ganesha cluster and recreate a new one Actual results: ======== Post upgrade, NFS Ganesha cluster deployment is failing Expected results: ========= NFS Ganesha deployment should Pass Additional info: ========= ganesha.log ---- Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :WARN :Use idmapped_group_time_validity under DIRECTORY_SERVICES section to configure time validity of idmapped groups Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] main :MAIN :WARN :Failed to set PR_SET_IO_FLUSHER due to EPERM, ignoring... Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Connection Manager initialized. Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] nfs4_recovery_init :CLIENT ID :EVENT :Recovery Backend Init for rados_cluster Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] set_nodeid :CLIENT ID :EVENT :Nodeid : 0 Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] rados_cluster_init :CLIENT ID :CRIT :Cluster membership check failed: -2 Jun 02 10:58:55 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:55 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] main :NFS STARTUP :CRIT :Recovery backend initialization failed! Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] main :NFS STARTUP :FATAL :Fatal errors. Server exiting... Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libganesha_nfsd.so.6.5(+0x9b9a1) [0x7f34405b09a1] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libganesha_nfsd.so.6.5(+0x99ed5) [0x7f34405aeed5] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libganesha_nfsd.so.6.5(DisplayLogComponentLevel+0x8b) [0x7f34405af2bb] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/usr/bin/ganesha.nfsd(main+0x71a) [0x559bed606eaa] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libc.so.6(+0x295d0) [0x7f34403165d0] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/lib64/libc.so.6(__libc_start_main+0x80) [0x7f3440316680] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz[145118]: 02/06/2025 10:58:56 : epoch 683d83ef : ceph-nfsupgradetest-fu6x8g-node2 : ganesha.nfsd-2[main] gsh_backtrace :NFS STARTUP :MAJ :/usr/bin/ganesha.nfsd(_start+0x25) [0x559bed607635] Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 podman[145158]: 2025-06-02 10:58:56.563333963 +0000 UTC m=+0.019873604 container died 04453b26a234d336f5d57f4288522b4fd3ed7953738cc5f971c20eceb729b27d (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:24c60de0b1a36b2250119e8487bf4ac5faf25d2e02fab36b3a48716e64e707f0, name=ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz, distribution-scope=public, release=475, RELEASE=main, io.openshift.tags=rhceph ceph, architecture=x86_64, GIT_CLEAN=True, com.redhat.license_terms=https://www.redhat.com/agreements, GIT_BRANCH=main, vendor=Red Hat, Inc., io.k8s.description=Red Hat Ceph Storage 8, maintainer=Guillaume Abrioux <gabrioux>, name=rhceph, ceph=True, version=8, com.redhat.component=rhceph-container, io.openshift.expose-services=, vcs-ref=900973db52a0d22b9fbdb24fa5fca76592c785b0, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-475, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, description=Red Hat Ceph Storage 8, build-date=2025-06-01T22:19:27, CEPH_POINT_RELEASE=, io.buildah.version=1.33.12, vcs-type=git) Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 podman[145158]: 2025-06-02 10:58:56.583621319 +0000 UTC m=+0.040160957 container remove 04453b26a234d336f5d57f4288522b4fd3ed7953738cc5f971c20eceb729b27d (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:24c60de0b1a36b2250119e8487bf4ac5faf25d2e02fab36b3a48716e64e707f0, name=ceph-b35b1132-3f95-11f0-9e8c-fa163e745294-nfs-nfsganesha-0-0-ceph-nfsupgradetest-fu6x8g-node2-xgdnwz, io.openshift.expose-services=, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-475, CEPH_POINT_RELEASE=, com.redhat.license_terms=https://www.redhat.com/agreements, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.component=rhceph-container, io.buildah.version=1.33.12, version=8, name=rhceph, ceph=True, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, GIT_CLEAN=True, description=Red Hat Ceph Storage 8, io.k8s.description=Red Hat Ceph Storage 8, maintainer=Guillaume Abrioux <gabrioux>, io.openshift.tags=rhceph ceph, vcs-type=git, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., distribution-scope=public, vcs-ref=900973db52a0d22b9fbdb24fa5fca76592c785b0, release=475, RELEASE=main, GIT_BRANCH=main, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, build-date=2025-06-01T22:19:27) Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: ceph-b35b1132-3f95-11f0-9e8c-fa163e745294.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 02 10:58:56 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: ceph-b35b1132-3f95-11f0-9e8c-fa163e745294.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz.service: Failed with result 'exit-code'. Jun 02 10:59:06 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: ceph-b35b1132-3f95-11f0-9e8c-fa163e745294.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz.service: Scheduled restart job, restart counter is at 5. Jun 02 10:59:06 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: Stopped Ceph nfs.nfsganesha.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz for b35b1132-3f95-11f0-9e8c-fa163e745294. Jun 02 10:59:06 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: ceph-b35b1132-3f95-11f0-9e8c-fa163e745294.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz.service: Start request repeated too quickly. Jun 02 10:59:06 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: ceph-b35b1132-3f95-11f0-9e8c-fa163e745294.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz.service: Failed with result 'exit-code'. Jun 02 10:59:06 ceph-nfsupgradetest-fu6x8g-node2 systemd[1]: Failed to start Ceph nfs.nfsganesha.0.0.ceph-nfsupgradetest-fu6x8g-node2.xgdnwz for b35b1132-3f95-11f0-9e8c-fa163e745294.