Bug 1901036

Summary:	[Ceph-dashboard] Default https based dashboard Ceph metric endpoint broken
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Sunil Angadi <sangadi>
Component:	Ceph-Mgr Plugins	Assignee:	Boris Ranto <branto>
Status:	CLOSED ERRATA	QA Contact:	Pawan <pdhiran>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.1	CC:	ceph-eng-bugs, ceph-qe-bugs, epuertat, jdillama, pdhiran, tserlin, vereddy
Target Milestone:	---	Keywords:	Regression
Target Release:	4.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-14.2.11-87.el8cp, ceph-14.2.11-87.el7cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-01-12 14:58:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 3 Boris Ranto 2020-11-24 13:28:51 UTC

I was looking at the cluster and I have a couple of notes, here:

Upstream back-ported this to nautilus (it is in 14.2.11) :

https://github.com/ceph/ceph/pull/35918

This means that we are now spawning a new thread in ceph-mgr prometheus module.

This does seem to cause issues when restarting the ceph-mgr daemon as it fails to collect the metrics in the thread after a restart.

I can get the module to work properly after a couple of (force) restarts. The module misbehaves after another restart (or two) though.

I am wondering if we should just revert the change in nautilus.

I am also wondering if this is reproducible in Octopus. Maybe, we are just missing some code that improves thread shutdown in nautilus?

Comment 4 Boris Ranto 2020-11-25 09:56:34 UTC

OK, some more details. The patch that I mentioned in my previous comment is actually hiding the real error. After I played with this a little more, the metrics collection fails because of

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/prometheus/module.py", line 194, in collect
    data = self.mod.collect()
  File "/usr/share/ceph/mgr/prometheus/module.py", line 977, in collect
    self.get_mgr_status()
  File "/usr/share/ceph/mgr/prometheus/module.py", line 546, in get_mgr_status
    ceph_release = host_version[1].split()[-2] # e.g. nautilus
IndexError: list index out of range

The host_version is defined as

host_version = servers.get((mgr, 'mgr'), ('', ''))

However, the code doesn't work for the default value as there is really nothing to split and it fails the whole metrics collection and eventually the whole metrics collection thread.

We only needed this to get the always on modules for the given ceph release. This is not something we should do anyway and we should instead join all the modules for all the releases.

This PR should fix this:

https://github.com/ceph/ceph/pull/38277

Comment 11 errata-xmlrpc 2021-01-12 14:58:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0081