[8.0][Bootstrap][Failing] Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
Description of problem:
[8.0][Bootstrap][Failing] Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
Version:
=========
19.1.1-8
Repo:
=====
RPM:http://download-01.beak-001.prod.iad2.dc.redhat.com/rhel-9/composes/auto/ceph-8.0-rhel-9/RHCEPH-8.0-RHEL-9-20240905.ci.4
Image: registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854
Logs:
=====
# rpm -qa | grep cephadm
cephadm-19.1.1-8.el9cp.noarch
# cephadm --image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 bootstrap --mon-ip 10.0.210.228
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 4.9.4 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 4935e2ee-6bf3-11ef-bcd7-fa163e054643
Verifying IP 10.0.210.228 port 3300 ...
Verifying IP 10.0.210.228 port 6789 ...
Mon IP `10.0.210.228` is in CIDR network `10.0.208.0/22`
Mon IP `10.0.210.228` is in CIDR network `10.0.208.0/22`
Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Logging into custom registry.
Pulling container image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854...
Ceph version: ceph version 19.1.1-8.el9cp (4ccb0885c36f17afa539cabf15c93152321a13c4) squid (rc)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 10.0.208.0/22 in mon config section
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Non-zero exit code 95 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm
/usr/bin/ceph: stderr Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
/usr/bin/ceph: stderr Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
***************
Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically.
To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous
broken installation, users must use the following command to completely delete the broken cluster:
> cephadm rm-cluster --force --zap-osds --fsid <fsid>
for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
***************
Deleting cluster with fsid: 4935e2ee-6bf3-11ef-bcd7-fa163e054643
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 5616, in <module>
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 5604, in main
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2655, in _rollback
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 449, in _default_image
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2904, in command_bootstrap
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2408, in enable_cephadm_mgr_module
File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2834, in cli
File "/tmp/tmpc16m67y2.cephadm.build/app/cephadmlib/container_types.py", line 429, in run
File "/tmp/tmpc16m67y2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
How reproducible:
==================
Always (3/3)
Steps to Reproduce:
====================
1. Bootstrap ceph cluster
Actual results:
Bootstrap failing
Expected results:
Bootstrap should pass
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2024:10216