Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2346107

Summary:	grafana.update.checker trying to reach Internet in a disconnected environment
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	John Fulton <johfulto>
Component:	Ceph-Dashboard	Assignee:	Nizamudeen <nia>
Status:	CLOSED ERRATA	QA Contact:	Sayalee <saraut>
Severity:	medium	Docs Contact:	Rivka Pollack <rpollack>
Priority:	unspecified
Version:	5.3	CC:	ceph-eng-bugs, cephqe-warriors, gfidente, mobisht, nia, rbruzzon
Target Milestone:	---
Target Release:	9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-20.1.0-84	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2026-01-29 06:53:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Fulton 2025-02-17 14:06:05 UTC

As per "1.3. Upgrading the Red Hat Ceph Storage cluster in a disconnected environment":

https://docs.redhat.com/en/documentation/red_hat_ceph_storage/7/html-single/upgrade_guide/index#upgrading-the-red-hat-ceph-storage-cluster-in-a-disconnected-environment_upgrade

We should be able to run Ceph in a disconnected environment. This was accomplished but the following unexpected errors related to grafana have been observed:

~~~
logger=grafana.update.checker t=2025-01-31T16:40:13.526395413Z level=error msg="Update check failed" error="failed to get stable version from grafana.com: Get \"https://grafana.com/api/grafana/versions/stable\": dial tcp 34.120.177.193:443: i/o timeout" duration=10.001814247s
logger=plugin.signature.key_retriever t=2025-01-31T16:40:13.550870805Z level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": dial tcp 34.120.177.193:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
logger=infra.usagestats t=2025-01-31T16:41:21.527086398Z level=info msg="Usage stats are ready to report"
logger=plugin.signature.key_retriever t=2025-01-31T16:41:23.553379671Z level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
We did modify */var/lib/ceph/c55d7b90-3910-46f7-90af-242cac2d11bc/grafana.control0001-naz91/etc/grafana/grafana.ini*
~~~

To workaround these issues, the following variables were included in the grafana.ini (test made in /var/lib/ceph/c55d7b90-3910-46f7-90af-242cac2d11bc/grafana.control0001-naz91/etc/grafana/grafana.ini)

~~~
[analytics]
check_for_updates = false
reporting_enabled = false
[plugins]
check_for_plugin_updates = false
public_key_retrieval_disabled = true
~~~

This workaround works but, this change isn't persistent and is lost (overrided) from the ceph dashboard restart:
~~~
ceph mgr module disable dashboard
ceph mgr module enable dashboard
~~~
Can a persistent way to solve this (without editing files in /var/lib) be supported and documented?

Comment 1 Storage PM bot 2025-02-17 14:06:13 UTC

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 2 John Fulton 2025-02-17 14:08:04 UTC

As per the cephadm grafana template code  [1] there isn't a simple way to apply a config-key to inject or populate this section. I think this is a valid bug / feature for the Ceph product in a disconnected environment. 

[1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/templates/services/grafana/grafana.ini.j2

Comment 3 John Fulton 2025-02-17 14:29:19 UTC

For context, this problem was discovered after the cluster was upgraded from 6 to 7

https://docs.redhat.com/en/documentation/red_hat_ceph_storage/7/html/upgrade_guide/upgrade-a-red-hat-ceph-storage-cluster-using-cephadm#upgrading-the-red-hat-ceph-storage-cluster_upgrade

Comment 12 errata-xmlrpc 2026-01-29 06:53:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2026:1536

Comment 13 Red Hat Bugzilla 2026-02-06 04:26:02 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days or the product is inactive and locked