Bug 2312501

Summary: [RHCS 8.0] [NFS-Ganesha] NFS-Ganesha mount with vers=3 is failing. Causing Ganesha service to crash
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.0CC: adking, akraj, cephqe-warriors, kkeithle, spunadik, tserlin
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-19.1.1-83.el9cp Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-11-25 09:10:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2348763, 2300272, 2317218    

Description Manisha Saini 2024-09-16 05:43:18 UTC 
Description of problem:
======================

Mount with vers=3 is failing, causing NFS ganesha to crash.

===
Sep 16 01:39:14 ceph-msaini-d0azq5-node2 ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb-nfs-nfsganesha-1-0-ceph-msaini-d0azq5-node2-nnfvgl[63204]: 16/09/2024 05:39:14 : epoch 66e73bbf : ceph-msaini-d0azq5-node2 : ganesha.nfsd-2[svc_914] rpc :TIRPC :EVENT :svc_dg_rendezvous: Bad message sa_family is 0xffff
Sep 16 01:39:14 ceph-msaini-d0azq5-node2 ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb-nfs-nfsganesha-1-0-ceph-msaini-d0azq5-node2-nnfvgl[63204]: 16/09/2024 05:39:14 : epoch 66e73bbf : ceph-msaini-d0azq5-node2 : ganesha.nfsd-2[svc_976] rpc :TIRPC :EVENT :svc_dg_rendezvous: Bad message sa_family is 0xffff
Sep 16 01:39:14 ceph-msaini-d0azq5-node2 ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb-nfs-nfsganesha-1-0-ceph-msaini-d0azq5-node2-nnfvgl[63204]: 16/09/2024 05:39:14 : epoch 66e73bbf : ceph-msaini-d0azq5-node2 : ganesha.nfsd-2[svc_866] rpc :TIRPC :EVENT :svc_dg_rendezvous: Bad message sa_family is 0xffff
Sep 16 01:39:14 ceph-msaini-d0azq5-node2 ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb-nfs-nfsganesha-1-0-ceph-msaini-d0azq5-node2-nnfvgl[63204]: 16/09/2024 05:39:14 : epoch 66e73bbf : ceph-msaini-d0azq5-node2 : ganesha.nfsd-2[svc_974] rpc :TIRPC :EVENT :svc_dg_rendezvous: Bad message sa_family is 0xffff
Sep 16 01:39:24 ceph-msaini-d0azq5-node2 systemd[1]: ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb.1.0.ceph-msaini-d0azq5-node2.nnfvgl.service: A process of this unit has been killed by the OOM killer.
Sep 16 01:39:25 ceph-msaini-d0azq5-node2 podman[279944]: 2024-09-16 01:39:25.842222107 -0400 EDT m=+0.142180141 container died cf3388a93979b650a032761fb254cce1c23760ee56f2a23b3c21512636ca06ff (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:ad6ba8caec2a8a3d5b98282677de2fc6d489ed6ad212668e5504d59387c83eea, name=ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb-nfs-nfsganesha-1-0-ceph-msaini-d0azq5-node2-nnfvgl, distribution-scope=public, name=rhceph, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, release=87, version=8, vendor=Red Hat, Inc., io.openshift.tags=rhceph ceph, CEPH_POINT_RELEASE=, vcs-type=git, ceph=True, build-date=2024-09-13T11:07:48, description=Red Hat Ceph Storage 8, io.buildah.version=1.29.0, com.redhat.license_terms=https://www.redhat.com/agreements, maintainer=Guillaume Abrioux <gabrioux>, summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., architecture=x86_64, io.openshift.expose-services=, GIT_CLEAN=True, RELEASE=main, com.redhat.component=rhceph-container, io.k8s.description=Red Hat Ceph Storage 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-87, GIT_BRANCH=main, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, vcs-ref=f0f2707c29c8affe98c484af48cf2d3b5459146f)
Sep 16 01:39:25 ceph-msaini-d0azq5-node2 podman[279944]: 2024-09-16 01:39:25.923415759 -0400 EDT m=+0.223373790 container remove cf3388a93979b650a032761fb254cce1c23760ee56f2a23b3c21512636ca06ff (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:ad6ba8caec2a8a3d5b98282677de2fc6d489ed6ad212668e5504d59387c83eea, name=ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb-nfs-nfsganesha-1-0-ceph-msaini-d0azq5-node2-nnfvgl, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/agreements, vendor=Red Hat, Inc., summary=Provides the latest Red Hat Ceph Storage 8 on RHEL 9 in a fully featured and supported base image., io.buildah.version=1.29.0, io.openshift.expose-services=, build-date=2024-09-13T11:07:48, io.k8s.description=Red Hat Ceph Storage 8, io.k8s.display-name=Red Hat Ceph Storage 8 on RHEL 9, com.redhat.component=rhceph-container, GIT_CLEAN=True, name=rhceph, version=8, maintainer=Guillaume Abrioux <gabrioux>, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/8-87, architecture=x86_64, CEPH_POINT_RELEASE=, release=87, description=Red Hat Ceph Storage 8, RELEASE=main, ceph=True, GIT_COMMIT=55ad0f204a1d654ee565abf874aecad0cc209d0e, io.openshift.tags=rhceph ceph, GIT_BRANCH=main, vcs-ref=f0f2707c29c8affe98c484af48cf2d3b5459146f, vcs-type=git)
Sep 16 01:39:25 ceph-msaini-d0azq5-node2 systemd[1]: ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb.1.0.ceph-msaini-d0azq5-node2.nnfvgl.service: Main process exited, code=exited, status=137/n/a
Sep 16 01:39:26 ceph-msaini-d0azq5-node2 systemd[1]: ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb.1.0.ceph-msaini-d0azq5-node2.nnfvgl.service: Failed with result 'exit-code'.
Sep 16 01:39:26 ceph-msaini-d0azq5-node2 systemd[1]: ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb.1.0.ceph-msaini-d0azq5-node2.nnfvgl.service: Consumed 2min 59.471s CPU time.
Sep 16 01:39:36 ceph-msaini-d0azq5-node2 systemd[1]: ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb.1.0.ceph-msaini-d0azq5-node2.nnfvgl.service: Scheduled restart job, restart counter is at 3.
Sep 16 01:39:36 ceph-msaini-d0azq5-node2 systemd[1]: Stopped Ceph nfs.nfsganesha.1.0.ceph-msaini-d0azq5-node2.nnfvgl for 2c8e7f60-7392-11ef-bf12-fa163e196fcb.
Sep 16 01:39:36 ceph-msaini-d0azq5-node2 systemd[1]: ceph-2c8e7f60-7392-11ef-bf12-fa163e196fcb.1.0.ceph-msaini-d0azq5-node2.nnfvgl.service: Consumed 2min 59.471s CPU time.
Sep 16 01:39:36 ceph-msaini-d0azq5-node2 systemd[1]: Starting Ceph nfs.nfsganesha.1.0.ceph-msaini-d0azq5-node2.nnfvgl for 2c8e7f60-7392-11ef-bf12-fa163e196fcb...
==========


Version-Release number of selected component (if applicable):
=================

# rpm -qa | grep nfs
libnfsidmap-2.5.4-26.el9_4.x86_64
nfs-utils-2.5.4-26.el9_4.x86_64
nfs-ganesha-selinux-6.0-4.el9cp.noarch
nfs-ganesha-6.0-4.el9cp.x86_64
nfs-ganesha-rgw-6.0-4.el9cp.x86_64
nfs-ganesha-ceph-6.0-4.el9cp.x86_64
nfs-ganesha-rados-grace-6.0-4.el9cp.x86_64
nfs-ganesha-rados-urls-6.0-4.el9cp.x86_64

# ceph --version
ceph version 19.1.1-39.el9cp (ff85d72be6c3acea30916eee3219ab627a3f9c15) squid (rc)


How reproducible:
================
2/2


Steps to Reproduce:
===================
1. Create NFS Ganesha cluster
2. Create an NFS export
3. Mount the export on client via v3

Actual results:
==============
Mount with vers=3 is failing causing NFS service crash


Expected results:
===============
Mount should work as expected


Additional info:
Comment 13 errata-xmlrpc 2024-11-25 09:10:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:10216