Bug 2310340

Description of problem:
[8.0][Bootstrap][Failing] Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it

Version:
=========
19.1.1-8

Repo:
=====
RPM:http://download-01.beak-001.prod.iad2.dc.redhat.com/rhel-9/composes/auto/ceph-8.0-rhel-9/RHCEPH-8.0-RHEL-9-20240905.ci.4
Image: registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 

Logs:
=====
# rpm -qa | grep cephadm
cephadm-19.1.1-8.el9cp.noarch

#  cephadm --image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 bootstrap --mon-ip 10.0.210.228
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 4.9.4 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 4935e2ee-6bf3-11ef-bcd7-fa163e054643
Verifying IP 10.0.210.228 port 3300 ...
Verifying IP 10.0.210.228 port 6789 ...
Mon IP `10.0.210.228` is in CIDR network `10.0.208.0/22`
Mon IP `10.0.210.228` is in CIDR network `10.0.208.0/22`
Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Logging into custom registry.
Pulling container image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854...
Ceph version: ceph version 19.1.1-8.el9cp (4ccb0885c36f17afa539cabf15c93152321a13c4) squid (rc)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 10.0.208.0/22 in mon config section
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Non-zero exit code 95 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm
/usr/bin/ceph: stderr Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
/usr/bin/ceph: stderr Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it



	***************
	Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically.
	To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous
	broken installation, users must use the following command to completely delete the broken cluster:

	> cephadm rm-cluster --force --zap-osds --fsid <fsid>

	for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
	***************


Deleting cluster with fsid: 4935e2ee-6bf3-11ef-bcd7-fa163e054643
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 5616, in <module>
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 5604, in main
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2655, in _rollback
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 449, in _default_image
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2904, in command_bootstrap
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2408, in enable_cephadm_mgr_module
  File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2834, in cli
  File "/tmp/tmpc16m67y2.cephadm.build/app/cephadmlib/container_types.py", line 429, in run
  File "/tmp/tmpc16m67y2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it


How reproducible:
==================
Always (3/3)

Steps to Reproduce:
====================
1. Bootstrap ceph cluster


Actual results:
Bootstrap failing

Expected results:
Bootstrap should pass