Description of problem: [8.0][Bootstrap][Failing] Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it Version: ========= 19.1.1-8 Repo: ===== RPM:http://download-01.beak-001.prod.iad2.dc.redhat.com/rhel-9/composes/auto/ceph-8.0-rhel-9/RHCEPH-8.0-RHEL-9-20240905.ci.4 Image: registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 Logs: ===== # rpm -qa | grep cephadm cephadm-19.1.1-8.el9cp.noarch # cephadm --image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 bootstrap --mon-ip 10.0.210.228 Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.9.4 is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 4935e2ee-6bf3-11ef-bcd7-fa163e054643 Verifying IP 10.0.210.228 port 3300 ... Verifying IP 10.0.210.228 port 6789 ... Mon IP `10.0.210.228` is in CIDR network `10.0.208.0/22` Mon IP `10.0.210.228` is in CIDR network `10.0.208.0/22` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Logging into custom registry. Pulling container image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854... Ceph version: ceph version 19.1.1-8.el9cp (4ccb0885c36f17afa539cabf15c93152321a13c4) squid (rc) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting public_network to 10.0.208.0/22 in mon config section Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 0.0.0.0:9283 ... Verifying port 0.0.0.0:8765 ... Verifying port 0.0.0.0:8443 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Non-zero exit code 95 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm /usr/bin/ceph: stderr Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date /usr/bin/ceph: stderr Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it *************** Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically. To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous broken installation, users must use the following command to completely delete the broken cluster: > cephadm rm-cluster --force --zap-osds --fsid <fsid> for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster *************** Deleting cluster with fsid: 4935e2ee-6bf3-11ef-bcd7-fa163e054643 Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 5616, in <module> File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 5604, in main File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2655, in _rollback File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 449, in _default_image File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2904, in command_bootstrap File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2408, in enable_cephadm_mgr_module File "/tmp/tmpc16m67y2.cephadm.build/app/__main__.py", line 2834, in cli File "/tmp/tmpc16m67y2.cephadm.build/app/cephadmlib/container_types.py", line 429, in run File "/tmp/tmpc16m67y2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 -e NODE_NAME=mobisht-test1 -v /var/log/ceph/4935e2ee-6bf3-11ef-bcd7-fa163e054643:/var/log/ceph:z -v /tmp/ceph-tmpumq14_0w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4m8zxk0s:/etc/ceph/ceph.conf:z registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.0-rhel-9-containers-candidate-35974-20240906010854 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it How reproducible: ================== Always (3/3) Steps to Reproduce: ==================== 1. Bootstrap ceph cluster Actual results: Bootstrap failing Expected results: Bootstrap should pass
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:10216