I was looking at the cluster and I have a couple of notes, here: Upstream back-ported this to nautilus (it is in 14.2.11) : https://github.com/ceph/ceph/pull/35918 This means that we are now spawning a new thread in ceph-mgr prometheus module. This does seem to cause issues when restarting the ceph-mgr daemon as it fails to collect the metrics in the thread after a restart. I can get the module to work properly after a couple of (force) restarts. The module misbehaves after another restart (or two) though. I am wondering if we should just revert the change in nautilus. I am also wondering if this is reproducible in Octopus. Maybe, we are just missing some code that improves thread shutdown in nautilus?
OK, some more details. The patch that I mentioned in my previous comment is actually hiding the real error. After I played with this a little more, the metrics collection fails because of Traceback (most recent call last): File "/usr/share/ceph/mgr/prometheus/module.py", line 194, in collect data = self.mod.collect() File "/usr/share/ceph/mgr/prometheus/module.py", line 977, in collect self.get_mgr_status() File "/usr/share/ceph/mgr/prometheus/module.py", line 546, in get_mgr_status ceph_release = host_version[1].split()[-2] # e.g. nautilus IndexError: list index out of range The host_version is defined as host_version = servers.get((mgr, 'mgr'), ('', '')) However, the code doesn't work for the default value as there is really nothing to split and it fails the whole metrics collection and eventually the whole metrics collection thread. We only needed this to get the always on modules for the given ceph release. This is not something we should do anyway and we should instead join all the modules for all the releases. This PR should fix this: https://github.com/ceph/ceph/pull/38277
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0081