Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use Jira Cloud for all bug tracking management.

Bug 2314311

Summary: [NFS-Ganesha] Could not send metrics, aborting: Broken pipe - ganesha.log
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: NFS-GaneshaAssignee: Sreedhar Agraharam <sragraha>
NFS-Ganesha sub component: Ceph QA Contact: Manisha Saini <msaini>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: cephqe-warriors, jcaratza, kkeithle, sragraha, tserlin, vdas, vthumuku
Version: 8.0   
Target Milestone: ---   
Target Release: 9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfs-ganesha-6.5-29.el9cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2026-01-29 06:52:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2024-09-23 20:33:18 UTC
Description of problem:
=====================

Create 10,000 files on the NFS mount point and load the 

1. ceph metrics output --> URL -> http://10.8.128.216:9587/metrics
2. prometheus": http://10.8.128.219:9283/metrics

Check ganesha.log
================
Sep 23 20:00:57 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: 23/09/2024 20:00:57 : epoch 66f1c8d8 : argo016 : ganesha.nfsd-2[reaper] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(4) clid count(5)
Sep 23 20:01:07 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: 23/09/2024 20:01:07 : epoch 66f1c8d8 : argo016 : ganesha.nfsd-2[reaper] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(5) clid count(5)
Sep 23 20:01:07 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: 23/09/2024 20:01:07 : epoch 66f1c8d8 : argo016 : ganesha.nfsd-2[reaper] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(5) clid count(5)
Sep 23 20:01:07 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: 23/09/2024 20:01:07 : epoch 66f1c8d8 : argo016 : ganesha.nfsd-2[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
Sep 23 20:03:13 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:03:13 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:03:13 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:03:13 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:03:51 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:03:52 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:26:47 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:26:47 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:26:47 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe
Sep 23 20:26:47 argo016 ceph-6146f1e0-bfb2-11ee-94e2-ac1f6b0a1844-nfs-nfs-0-0-argo016-qovwrk[1225242]: [/builddir/build/BUILD/nfs-ganesha-6.0/src/monitoring/exposer.cc:87] Could not send metrics, aborting: Broken pipe


Version-Release number of selected component (if applicable):
=================
# ceph --version
ceph version 19.1.1-68.el9cp (f12a1df704fe62bc41993073af0d9077e157325d) squid (rc)

# rpm -qa | grep nfs
libnfsidmap-2.5.4-26.el9_4.x86_64
nfs-utils-2.5.4-26.el9_4.x86_64
nfs-ganesha-selinux-6.0-5.el9cp.noarch
nfs-ganesha-6.0-5.el9cp.x86_64
nfs-ganesha-rgw-6.0-5.el9cp.x86_64
nfs-ganesha-ceph-6.0-5.el9cp.x86_64
nfs-ganesha-rados-grace-6.0-5.el9cp.x86_64
nfs-ganesha-rados-urls-6.0-5.el9cp.x86_64


How reproducible:


Steps to Reproduce:
1.Create ganesha cluster
2. Create an NFS exports
3. Mount the export on client and create 10.000 files on NFS mount point
# for i in $(seq 1 10000);do mkdir directory$i;done
4. Reload the ceph metrics and prometheus metrics page


Actual results:
================
"Could not send metrics, aborting: Broken pipe" messages were observed in ganesha.log


Expected results:
================
No errors should be seen


Additional info:

Comment 2 Sreedhar Agraharam 2025-03-27 05:53:56 UTC
Hi @manisha,

Tried with the standalone nfs-ganesha service.

I am not able to reproduce the issue as mentioned.

Now, I Will try with ceph build.

Comment 3 Sreedhar Agraharam 2025-03-27 05:54:42 UTC
Hi @manisha,

Tried with the standalone nfs-ganesha service.

I am not able to reproduce the issue as mentioned.

Now, I Will try with ceph build.

Comment 4 Vidya Thumukunta 2025-04-03 11:17:03 UTC
Hi @Manisha

I have tested the above scenario in Ganesha cluster.
Followed the reproducible steps, created 10,000 directories on mount point, and metrics are being updated in the prometheus metrics page.

No errors related to metrics are found in Ganesha.log.

Please retest and confirm.

Comment 7 Sreedhar Agraharam 2025-08-05 14:00:29 UTC
Hi @manisha

I tested using the following Ceph downstream branch and version:

Note: Used vstart to bring up the environment.
# Beanch info
root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ceph/build# git branch
* ceph-8.1z2-rhel-patches-STAGING
  master

#Ceph version
root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ceph/build# bin/ceph --version
ceph version 19.2.1-1824-g26e9d112cae (26e9d112caed559f683ef192136ea32cdfc04f27) squid (stable)
root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ceph/build# 
=========================
I checked for any "Broken" messages in the Ganesha log but found none:
root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ceph/build# cat /home/sreedhar/ceph_build/downstream/ceph/build/out/ganesha-a.log  | grep Broken
root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ceph/build# 
=========================
On the client side:
[root@rhs-srv-09-CentOS9-sreedhar-vm2 sreedhar]# mount -t nfs -o port=44435 10.8.159.61:/cephfs check_me/
[root@rhs-srv-09-CentOS9-sreedhar-vm2 sreedhar]# 
[root@rhs-srv-09-CentOS9-sreedhar-vm2 sreedhar]# mount | grep check_me
10.8.159.61:/cephfs on /home/sreedhar/check_me type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=44435,timeo=600,retrans=2,sec=sys,clientaddr=10.8.159.216,local_lock=none,addr=10.8.159.61)
[root@rhs-srv-09-CentOS9-sreedhar-vm2 sreedhar]# cd check_me/
[root@rhs-srv-09-CentOS9-sreedhar-vm2 check_me]# 
[root@rhs-srv-09-CentOS9-sreedhar-vm2 check_me]# mkdir 1000_dirs
[root@rhs-srv-09-CentOS9-sreedhar-vm2 check_me]# cd 1000_dirs/
[root@rhs-srv-09-CentOS9-sreedhar-vm2 1000_dirs]# for i in $(seq 1 10000);do mkdir directory$i;done
[root@rhs-srv-09-CentOS9-sreedhar-vm2 1000_dirs]# ls -al | wc -l
10003
[root@rhs-srv-09-CentOS9-sreedhar-vm2 1000_dirs]# 
=========================
Ganesha branch and version

root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ganesha/nfs-ganesha# git branch
* ceph-9.0-rhel-patches
  master
root@rhs-srv-09-CentOS9-sreedhar-vm1:/home/sreedhar/ceph_build/downstream/ganesha/nfs-ganesha# 


=========================
Despite these operations, the metrics are still being displayed successfully on the HTML page:: http://10.8.159.61:9587/

=========================

Do I need to test with any branch other than ceph-8.1z2-rhel-patches-STAGING to reproduce the issue manisha?

Comment 26 errata-xmlrpc 2026-01-29 06:52:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2026:1536