Description of problem:
In OSP-17 RHEL-9 promotion pipeline, Ceph related jobs are failing while running cephadm bootstrap with following errors:
```
2023-06-19 01:01:16.383 | "2023-06-19 01:01:11.797284 | 525400ff-71a7-08f5-b3cd-000000000070 | FATAL | Run cephadm bootstrap | controller-0 | error={\"changed\": true, \"cmd\": \"/usr/sbin/cephadm --image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:6-115 \\\\bootstrap --yes-i-know --skip-firewalld --skip-prepare-host --ssh-private-key /home/ceph-admin/.ssh/id_rsa --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub --ssh-user ceph-admin --allow-fqdn-hostname --output-keyring /etc/ceph/ceph.client.admin.keyring --output-config /etc/ceph/ceph.conf --fsid a5b6afac-306c-535a-83c4-7f7b92297b52 --config /home/ceph-admin/assimilate_ceph.conf \\\\--skip-monitoring-stack --skip-dashboard --mon-ip fd00:fd00:fd00:3000::43\\n\", \"delta\": \"0:00:12.571894\", \"end\": \"2023-06-19 01:01:11.759579\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2023-06-19 01:00:59.187685\", \"stderr\": \"Skip prepare_host\\nCluster fsid: a5b6afac-306c-535a-83c4-7f7b92297b52\\nVerifying IP [fd00:fd00:fd00:3000::43] port 3300 ...\\nVerifying IP [fd00:fd00:fd00:3000::43] port 6789 ...\\nMon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\\nMon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\\nInternal network (--cluster-network) has not been provided, OSD replication will default to the public_network\\nPulling container image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:6-115...\\nCeph version: ceph version 17.2.5-75.el9cp (52c8ab07f1bc5423199eeb6ab5714bc30a930955) quincy (stable)\\nERROR: Container release quincy != cephadm release pacific; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)\", \"stderr_lines\": [\"Skip prepare_host\", \"Cluster fsid: a5b6afac-306c-535a-83c4-7f7b92297b52\", \"Verifying IP [fd00:fd00:fd00:3000::43] port 3300 ...\", \"Verifying IP [fd00:fd00:fd00:3000::43] port 6789 ...\", \"Mon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\", \"Mon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\", \"Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network\", \"Pulling container image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:6-115...\", \"Ceph version: ceph version 17.2.5-75.el9cp (52c8ab07f1bc5423199eeb6ab5714bc30a930955) quincy (stable)\", \"ERROR: Container release quincy != cephadm release pacific; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)\"], \"stdout\": \"\", \"stdout_lines\": []}",
```
It might be using older version of cephadm. that's why this issue is coming.
Adding this bug to track the issue.
I believe the problem is that we're deploying Quincy (RHCS6) containers using an old version of cephadm from Pacific (RHCS5). The cephadm package is installed on the overcloud nodes and it should be updated to the version shipping with Quincy.
Description of problem: In OSP-17 RHEL-9 promotion pipeline, Ceph related jobs are failing while running cephadm bootstrap with following errors: ``` 2023-06-19 01:01:16.383 | "2023-06-19 01:01:11.797284 | 525400ff-71a7-08f5-b3cd-000000000070 | FATAL | Run cephadm bootstrap | controller-0 | error={\"changed\": true, \"cmd\": \"/usr/sbin/cephadm --image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:6-115 \\\\bootstrap --yes-i-know --skip-firewalld --skip-prepare-host --ssh-private-key /home/ceph-admin/.ssh/id_rsa --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub --ssh-user ceph-admin --allow-fqdn-hostname --output-keyring /etc/ceph/ceph.client.admin.keyring --output-config /etc/ceph/ceph.conf --fsid a5b6afac-306c-535a-83c4-7f7b92297b52 --config /home/ceph-admin/assimilate_ceph.conf \\\\--skip-monitoring-stack --skip-dashboard --mon-ip fd00:fd00:fd00:3000::43\\n\", \"delta\": \"0:00:12.571894\", \"end\": \"2023-06-19 01:01:11.759579\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2023-06-19 01:00:59.187685\", \"stderr\": \"Skip prepare_host\\nCluster fsid: a5b6afac-306c-535a-83c4-7f7b92297b52\\nVerifying IP [fd00:fd00:fd00:3000::43] port 3300 ...\\nVerifying IP [fd00:fd00:fd00:3000::43] port 6789 ...\\nMon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\\nMon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\\nInternal network (--cluster-network) has not been provided, OSD replication will default to the public_network\\nPulling container image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:6-115...\\nCeph version: ceph version 17.2.5-75.el9cp (52c8ab07f1bc5423199eeb6ab5714bc30a930955) quincy (stable)\\nERROR: Container release quincy != cephadm release pacific; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)\", \"stderr_lines\": [\"Skip prepare_host\", \"Cluster fsid: a5b6afac-306c-535a-83c4-7f7b92297b52\", \"Verifying IP [fd00:fd00:fd00:3000::43] port 3300 ...\", \"Verifying IP [fd00:fd00:fd00:3000::43] port 6789 ...\", \"Mon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\", \"Mon IP `[fd00:fd00:fd00:3000::43]` is in CIDR network `fd00:fd00:fd00:3000::/64`\", \"Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network\", \"Pulling container image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:6-115...\", \"Ceph version: ceph version 17.2.5-75.el9cp (52c8ab07f1bc5423199eeb6ab5714bc30a930955) quincy (stable)\", \"ERROR: Container release quincy != cephadm release pacific; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)\"], \"stdout\": \"\", \"stdout_lines\": []}", ``` It might be using older version of cephadm. that's why this issue is coming. Adding this bug to track the issue.