Bug 2369533

Summary:	[8.0z5] Bootstrap is failing - Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'cephadm.services.service_registry'
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Manisha Saini <msaini>
Component:	Cephadm	Assignee:	Shweta Bhosale <shbhosal>
Status:	CLOSED ERRATA	QA Contact:	Manisha Saini <msaini>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	adking, cephqe-warriors, jcaratza, shbhosal, tserlin, vereddy
Target Milestone:	---	Keywords:	Automation, Regression
Target Release:	8.0z5
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-19.2.0-139.el9cp	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2025-06-09 14:17:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2365634, 2369412, 2369465

Description Manisha Saini 2025-05-31 02:45:49 UTC

Description of problem:
=======================

Bootstrap is failing with 8.0z5 builds with -- 
"Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'cephadm.services.service_registry'"



# cephadm --image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 bootstrap --mon-ip 10.0.66.215
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 5.2.2 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 58f3e0ee-3dc8-11f0-9ffc-fa163ef6587c
Verifying IP 10.0.66.215 port 3300 ...
Verifying IP 10.0.66.215 port 6789 ...
Mon IP `10.0.66.215` is in CIDR network `10.0.64.0/22`
Mon IP `10.0.66.215` is in CIDR network `10.0.64.0/22`
Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736...
Ceph version: ceph version 19.2.0-138.el9cp (50c92a296956d04ee96bab288a68e71dfe3ddda9) squid (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 10.0.64.0/22 in global config section
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr not available, waiting (4/15)...
mgr is available
Enabling cephadm module...
Non-zero exit code 2 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 -e NODE_NAME=ceph-nfsupgradetest-t26x3e-node1-installer -v /var/log/ceph/58f3e0ee-3dc8-11f0-9ffc-fa163ef6587c:/var/log/ceph:z -v /tmp/ceph-tmp6mt8et5k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph4rzzpcg:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 mgr module enable cephadm
/usr/bin/ceph: stderr Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'cephadm.services.service_registry' (pass --force to force enablement)
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 -e NODE_NAME=ceph-nfsupgradetest-t26x3e-node1-installer -v /var/log/ceph/58f3e0ee-3dc8-11f0-9ffc-fa163ef6587c:/var/log/ceph:z -v /tmp/ceph-tmp6mt8et5k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph4rzzpcg:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'cephadm.services.service_registry' (pass --force to force enablement)



	***************
	Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically.
	To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous
	broken installation, users must use the following command to completely delete the broken cluster:

	> cephadm rm-cluster --force --zap-osds --fsid <fsid>

	for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
	***************


Deleting cluster with fsid: 58f3e0ee-3dc8-11f0-9ffc-fa163ef6587c
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/sbin/cephadm/__main__.py", line 5989, in <module>
  File "/usr/sbin/cephadm/__main__.py", line 5977, in main
  File "/usr/sbin/cephadm/__main__.py", line 2702, in _rollback
  File "/usr/sbin/cephadm/__main__.py", line 453, in _default_image
  File "/usr/sbin/cephadm/__main__.py", line 3060, in command_bootstrap
  File "/usr/sbin/cephadm/__main__.py", line 2441, in enable_cephadm_mgr_module
  File "/usr/sbin/cephadm/__main__.py", line 2978, in cli
  File "/usr/sbin/cephadm/cephadmlib/container_types.py", line 429, in run
  File "/usr/sbin/cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 -e NODE_NAME=ceph-nfsupgradetest-t26x3e-node1-installer -v /var/log/ceph/58f3e0ee-3dc8-11f0-9ffc-fa163ef6587c:/var/log/ceph:z -v /tmp/ceph-tmp6mt8et5k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph4rzzpcg:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-84040-20250530204736 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'cephadm.services.service_registry' (pass --force to force enablement)


Version-Release number of selected component (if applicable):
===========

# rpm -qa | grep cephadm
cephadm-19.2.0-138.el9cp.noarch


How reproducible:
=======
2/2


Steps to Reproduce:
=======
1. Bootstrap cluster with latest 8.0z5 builds


Actual results:
=======
Bootstrap is failing - No module named 'cephadm.services.service_registry


Expected results:
=======
ceph deployment should pass


Additional info:

Comment 12 errata-xmlrpc 2025-06-09 14:17:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 bug fix updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2025:8694