I was looking at the cluster and I have a couple of notes, here:
Upstream back-ported this to nautilus (it is in 14.2.11) :
This means that we are now spawning a new thread in ceph-mgr prometheus module.
This does seem to cause issues when restarting the ceph-mgr daemon as it fails to collect the metrics in the thread after a restart.
I can get the module to work properly after a couple of (force) restarts. The module misbehaves after another restart (or two) though.
I am wondering if we should just revert the change in nautilus.
I am also wondering if this is reproducible in Octopus. Maybe, we are just missing some code that improves thread shutdown in nautilus?
OK, some more details. The patch that I mentioned in my previous comment is actually hiding the real error. After I played with this a little more, the metrics collection fails because of
Traceback (most recent call last):
File "/usr/share/ceph/mgr/prometheus/module.py", line 194, in collect
data = self.mod.collect()
File "/usr/share/ceph/mgr/prometheus/module.py", line 977, in collect
File "/usr/share/ceph/mgr/prometheus/module.py", line 546, in get_mgr_status
ceph_release = host_version.split()[-2] # e.g. nautilus
IndexError: list index out of range
The host_version is defined as
host_version = servers.get((mgr, 'mgr'), ('', ''))
However, the code doesn't work for the default value as there is really nothing to split and it fails the whole metrics collection and eventually the whole metrics collection thread.
We only needed this to get the always on modules for the given ceph release. This is not something we should do anyway and we should instead join all the modules for all the releases.
This PR should fix this:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.