Bug 2420485
| Summary: | ODF Internal mode cluster deployment fails with MGR HEALTH_ERR Module prometheus crash | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Elad <ebenahar> |
| Component: | Ceph-Dashboard | Assignee: | Nizamudeen <nia> |
| Status: | CLOSED ERRATA | QA Contact: | Vinayak Papnoi <vpapnoi> |
| Severity: | urgent | Docs Contact: | Rivka Pollack <rpollack> |
| Priority: | unspecified | ||
| Version: | 9.0 | CC: | afrahman, ceph-eng-bugs, cephqe-warriors, nia, tserlin, vereddy |
| Target Milestone: | --- | ||
| Target Release: | 9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-20.1.0-122 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2026-01-29 07:04:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2026:1536 |
Description of problem: ODF 4.21, Ceph9 based, Internal mode cluster deployment fails with the following crash: { "backtrace": [ " File \"/usr/share/ceph/mgr/prometheus/module.py\", line 1889, in configure\n security_config = json.loads(out)", " File \"/usr/lib64/python3.9/json/__init__.py\", line 346, in loads\n return _default_decoder.decode(s)", " File \"/usr/lib64/python3.9/json/decoder.py\", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())", " File \"/usr/lib64/python3.9/json/decoder.py\", line 355, in raw_decode\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None", "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)", "\nDuring handling of the above exception, another exception occurred:\n", "Traceback (most recent call last):", " File \"/usr/share/ceph/mgr/prometheus/module.py\", line 2048, in serve\n self.configure(server_addr, server_port)", " File \"/usr/share/ceph/mgr/prometheus/module.py\", line 1894, in configure\n self.log.exception(f'Failed to setup cephadm based secure monitoring stack: {e}\\n',", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 1481, in exception\n self.error(msg, *args, exc_info=exc_info, **kwargs)", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 1475, in error\n self._log(ERROR, msg, args, **kwargs)", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 1589, in _log\n self.handle(record)", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 1599, in handle\n self.callHandlers(record)", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 1661, in callHandlers\n hdlr.handle(record)", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 952, in handle\n self.emit(record)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 722, in emit\n self._module._ceph_log(self.format(record))", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 927, in format\n return fmt.format(record)", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 663, in format\n record.message = record.getMessage()", " File \"/usr/lib64/python3.9/logging/__init__.py\", line 367, in getMessage\n msg = msg % self.args", "TypeError: not all arguments converted during string formatting" ], "ceph_version": "20.1.0-117.el9cp", "crash_id": "2025-12-04T06:49:25.981273Z_4bac8bd0-5df7-42af-aab7-67ca487c9a8a", "entity_name": "mgr.a", "mgr_module": "prometheus", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "TypeError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.7 (Plow)", "os_version_id": "9.7", "process_name": "ceph-mgr", "stack_sig": "2b55a95864ec04efbf58dc7afa6b8642f682f0f937d7bef4be4da7f8d78ff856", "timestamp": "2025-12-04T06:49:25.981273Z", "utsname_hostname": "rook-ceph-mgr-a-5c6596747f-6c4h9", "utsname_machine": "x86_64", "utsname_release": "5.14.0-570.66.1.el9_6.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Mon Nov 17 08:39:18 EST 2025" } Version-Release number of selected component (if applicable): ceph version 20.1.0-120.el9cp (810391a4e0bcc3734d005b357fbe501331ed1c8d) tentacle (rc - RelWithDebInfo) ODF 4.21.0-43.konflux How reproducible: 100% Steps to Reproduce: 1. Deploy ODF 4.21 Ceph9 based Internal mode