Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2405962

Summary: [NFS-Ganesha] Prometheus cannot access NFS exporter metrics endpoints; NFS metrics URLs are unreachable
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: CephadmAssignee: Shweta Bhosale <shbhosal>
Status: CLOSED UPSTREAM QA Contact: Vinayak Papnoi <vpapnoi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 9.0CC: adking, cephqe-warriors, ngangadh, rpollack, shbhosal, sragraha
Target Milestone: ---   
Target Release: 9.0z1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
.Prometheus cannot access NFS exporter metrics endpoint Prometheus cannot access the NFS exporter metrics endpoint because the NFS monitoring port is not opened in the firewall rules. As a result, NFS metrics URLs are unreachable. As a workaround, open the NFS monitoring port in the firewall. The default NFS monitoring port is `9587`.
 Story Points: ---
Clone Of:
: 2421707 (view as bug list) Environment:
Last Closed: 2026-03-04 09:56:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2388233, 2421707    

Description Manisha Saini 2025-10-23 01:10:26 UTC 
Description of problem:
==============
Hitting this issue while verifying BZ - https://bugzilla.redhat.com/show_bug.cgi?id=2314311 .

Reported under "NFS-Ganesha" component initially. 

URL - http://10.0.66.138:9095/targets?search=

After deploying two NFS clusters on separate nodes, none of the NFS exporter endpoints (:9587) are reachable from the Prometheus node.

The Prometheus Targets page (http://<prometheus-host>:9095/targets) shows one NFS target in DOWN state with errors such as “connect: no route to host” 

Accessing the NFS exporter metrics URLs directly also fails.

The issue occurs even though the NFS-Ganesha services appear healthy from the Ceph side (ceph orch ps shows them as running).
On the Targets page, one NFS endpoint is reported as UP and the other as DOWN, but in reality, both URLs are inaccessible.


Version-Release number of selected component (if applicable):
==================================
# ceph --version
ceph version 20.1.0-64.el10cp (0fb3867d2d68f6f57325c3bd1d921c1f8452e930) tentacle (rc)


How reproducible:
==============
1/1


Steps to Reproduce:
====================
1.Deploy 2 nfs ganesha cluster on 2 different node
2.Check the NFS target endpoints from prometheus page 


Actual results:
===============
Metrics for the NFS endpoint target are not collected/URL not accessible for both the nfs node.One NFS endpoint as UP and Another NFS endpoint as DOWN


Expected results:
================
All NFS endpoints (http://<node>:9587/metrics) should be reachable from Prometheus.
/targets page should show all NFS instances in UP state.

No “connect: no route to host” errors should appear in Prometheus logs.


Additional info:
================
[ceph: root@ceph-msaini-1mj2bt-node1-installer /]# ceph mgr services
{
    "dashboard": "https://10.0.66.138:8443/",
    "prometheus": "http://10.0.66.138:9283/"
}


[ceph: root@ceph-msaini-1mj2bt-node1-installer /]# ceph orch host ls
HOST                                ADDR         LABELS                                                                            STATUS
ceph-msaini-1mj2bt-node1-installer  10.0.66.138  _admin,installer,osd,crash,prometheus,mgr,grafana,alertmanager,mon,node-exporter
ceph-msaini-1mj2bt-node2            10.0.64.80   osd,crash,mds,mgr,rgw,nfs,alertmanager,mon,node-exporter
ceph-msaini-1mj2bt-node3            10.0.64.188  osd,crash,mds,rgw,mon,node-exporter
3 hosts in cluster


# ceph nfs cluster ls | grep nfs
  "mynfs",
  "mynfs1"

[ceph: root@ceph-msaini-1mj2bt-node1-installer /]# ceph orch ls | grep nfs
nfs.mynfs                                   1/1  80s ago    3h   ceph-msaini-1mj2bt-node1-installer
nfs.mynfs1                                  1/1  9m ago     3h   ceph-msaini-1mj2bt-node2

[ceph: root@ceph-msaini-1mj2bt-node1-installer /]# ceph orch ps | grep nfs
nfs.mynfs1.0.0.ceph-msaini-1mj2bt-node2.ceagjp           ceph-msaini-1mj2bt-node2            *:2049            running (3h)     9m ago   3h    88.6M        -  7.0               86e845c04ed5  8c8f36cb25dc
nfs.mynfs.0.0.ceph-msaini-1mj2bt-node1-installer.ulotgv  ceph-msaini-1mj2bt-node1-installer  *:2049            running (3h)    84s ago   3h     172M        -  7.0               86e845c04ed5  096aa9fb90fb

[ceph: root@ceph-msaini-1mj2bt-node1-installer /]# ceph orch ps | grep prome
prometheus.ceph-msaini-1mj2bt-node1-installer            ceph-msaini-1mj2bt-node1-installer  *:9095            running (3h)     4s ago   5h     110M        -  2.51.0            1d3b7f56885b  e7d4df1b3bc9


Logs -
=======
Oct 23 00:22:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:22:01.809Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=ceph\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:22:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:22:01.810Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:22:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:22:01.811Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:22:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:22:01.812Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:22:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:22:01.812Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node-exporter msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:23:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:23:01.810Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=ceph\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:23:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:23:01.810Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:23:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:23:01.811Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:23:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:23:01.811Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node-exporter msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 10.0.64.80:8765: connect: connection refused"
Oct 23 00:23:01 ceph-msaini-1mj2bt-node1-installer ceph-34072a3e-af7e-11f0-a7ba-fa163e9e92e2-prometheus-ceph-msaini-1mj2bt-node1-installer[103948]: ts=2025-10-23T00:23:01.811Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://10.0.64.80:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 10.0.64.80:8765: connect: connection refused"
Comment 1 Storage PM bot 2025-10-23 01:10:35 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 10 Red Hat Bugzilla 2026-03-04 09:56:15 UTC 
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.