Description of problem: ======================= While performing upgrade from IBM 5.3 to IBM 6.1 , the preflight playbook fails with the below error: ---- snip ---- TASK [set_fact _ceph_repo] ***************************************************************************************************************** Wednesday 21 June 2023 10:57:26 -0400 (0:00:00.079) 0:00:00.267 ******** ok: [10.0.208.243] ok: [10.0.210.237] ok: [10.0.208.180] TASK [configure ceph repository key] ******************************************************************************************************* Wednesday 21 June 2023 10:57:27 -0400 (0:00:00.054) 0:00:00.321 ******** ok: [10.0.208.180] ok: [10.0.208.243] ok: [10.0.210.237] TASK [configure ceph stable repository] **************************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:01.064) 0:00:01.386 ******** changed: [10.0.208.180] => (item=$basearch) changed: [10.0.208.243] => (item=$basearch) changed: [10.0.210.237] => (item=$basearch) TASK [fetch ceph development repository] *************************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.383) 0:00:01.769 ******** skipping: [10.0.208.243] skipping: [10.0.210.237] skipping: [10.0.208.180] TASK [configure ceph development repository] *********************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.062) 0:00:01.831 ******** skipping: [10.0.208.243] skipping: [10.0.210.237] skipping: [10.0.208.180] TASK [remove ceph_stable repositories] ***************************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.040) 0:00:01.872 ******** skipping: [10.0.208.243] => (item=ceph_stable) skipping: [10.0.208.243] => (item=ceph_stable_noarch) skipping: [10.0.208.243] skipping: [10.0.208.180] => (item=ceph_stable) skipping: [10.0.208.180] => (item=ceph_stable_noarch) skipping: [10.0.208.180] skipping: [10.0.210.237] => (item=ceph_stable) skipping: [10.0.210.237] => (item=ceph_stable_noarch) skipping: [10.0.210.237] TASK [set_fact ceph_custom_repositories] *************************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.063) 0:00:01.936 ******** skipping: [10.0.208.243] skipping: [10.0.210.237] skipping: [10.0.208.180] TASK [setup custom repositories] *********************************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.038) 0:00:01.974 ******** skipping: [10.0.208.243] skipping: [10.0.210.237] skipping: [10.0.208.180] TASK [enable required CentOS repository for epel] ****************************************************************************************** Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.044) 0:00:02.019 ******** skipping: [10.0.208.243] skipping: [10.0.210.237] skipping: [10.0.208.180] TASK [install package] ********************************************************************************************************************* Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.042) 0:00:02.062 ******** skipping: [10.0.208.243] skipping: [10.0.210.237] skipping: [10.0.208.180] TASK [install ceph-common on rhel] ********************************************************************************************************* Wednesday 21 June 2023 10:57:28 -0400 (0:00:00.041) 0:00:02.104 ******** FAILED - RETRYING: [10.0.208.180]: install ceph-common on rhel (3 retries left). FAILED - RETRYING: [10.0.208.243]: install ceph-common on rhel (3 retries left). FAILED - RETRYING: [10.0.210.237]: install ceph-common on rhel (3 retries left). FAILED - RETRYING: [10.0.208.180]: install ceph-common on rhel (2 retries left). FAILED - RETRYING: [10.0.208.243]: install ceph-common on rhel (2 retries left). FAILED - RETRYING: [10.0.210.237]: install ceph-common on rhel (2 retries left). FAILED - RETRYING: [10.0.208.180]: install ceph-common on rhel (1 retries left). FAILED - RETRYING: [10.0.208.243]: install ceph-common on rhel (1 retries left). FAILED - RETRYING: [10.0.210.237]: install ceph-common on rhel (1 retries left). fatal: [10.0.208.180]: FAILED! => changed=false attempts: 3 cmd: - dnf - install - --allowerasing - --assumeyes - ceph-common delta: '0:00:01.419010' end: '2023-06-21 10:57:50.330223' msg: non-zero return code rc: 1 start: '2023-06-21 10:57:48.911213' stderr: |- Errors during downloading metadata for repository 'ceph_stable_x86_64': - Status code: 404 for https://public.dhe.ibm.com/ibmdl/export/pub/storage/ceph/6/rhel9//x86_64/repodata/repomd.xml (IP: 170.225.126.18) Error: Failed to download metadata for repo 'ceph_stable_x86_64': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried stderr_lines: <omitted> stdout: |- Updating Subscription Management repositories. This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions. IBM Ceph repo - x86_64 778 B/s | 196 B 00:00 stdout_lines: <omitted> fatal: [10.0.208.243]: FAILED! => changed=false attempts: 3 cmd: - dnf - install - --allowerasing - --assumeyes - ceph-common delta: '0:00:01.427132' end: '2023-06-21 10:57:50.604614' msg: non-zero return code rc: 1 start: '2023-06-21 10:57:49.177482' stderr: |- Errors during downloading metadata for repository 'ceph_stable_x86_64': - Status code: 404 for https://public.dhe.ibm.com/ibmdl/export/pub/storage/ceph/6/rhel9//x86_64/repodata/repomd.xml (IP: 170.225.126.18) Error: Failed to download metadata for repo 'ceph_stable_x86_64': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried stderr_lines: <omitted> stdout: |- Updating Subscription Management repositories. This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions. IBM Ceph repo - x86_64 777 B/s | 196 B 00:00 stdout_lines: <omitted> fatal: [10.0.210.237]: FAILED! => changed=false attempts: 3 cmd: - dnf - install - --allowerasing - --assumeyes - ceph-common delta: '0:00:01.740274' end: '2023-06-21 10:57:52.004864' msg: non-zero return code rc: 1 start: '2023-06-21 10:57:50.264590' stderr: |- Errors during downloading metadata for repository 'ceph_stable_x86_64': - Status code: 404 for https://public.dhe.ibm.com/ibmdl/export/pub/storage/ceph/6/rhel9//x86_64/repodata/repomd.xml (IP: 170.225.126.18) Error: Failed to download metadata for repo 'ceph_stable_x86_64': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried stderr_lines: <omitted> stdout: |- Updating Subscription Management repositories. This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions. IBM Ceph repo - x86_64 703 B/s | 196 B 00:00 stdout_lines: <omitted> PLAY RECAP ********************************************************************************************************************************* 10.0.208.180 : ok=3 changed=1 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 10.0.208.243 : ok=3 changed=1 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 10.0.210.237 : ok=3 changed=1 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 localhost : ok=0 changed=0 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 Wednesday 21 June 2023 10:57:52 -0400 (0:00:23.245) 0:00:25.349 ******** ---- snip ---- Version-Release number of selected component (if applicable): ============================================================= # rpm -qa | grep cephadm cephadm-17.2.6-70.el9cp.noarch cephadm-ansible-2.15.0-1.el9cp.noarch How reproducible: 1/1 Steps to Reproduce: =================== 1. Deploy IBM ceph 5.3 cluster 2. Perform host OS upgrade 3. Update cephadm packages and then run the cephadm-preflight.yml playbook Actual results: =============== The preflight playbook fails with the error mentioned above. Expected results: ================= Preflight playbook must succeed and the upgrade should also be successful Additional info: ================ While the playbook is running, there is a new "ceph-stable" repo created. The contents of the repo are as follows: [root@ceph-vpapnoi-ibm-hmmiwn-node1-installer yum.repos.d]# pwd /etc/yum.repos.d [root@ceph-vpapnoi-ibm-hmmiwn-node1-installer yum.repos.d]# ls 'ceph_stable_$basearch.repo' ibm-storage-ceph-6-rhel-9.repo lab-extras.repo redhat.repo [root@ceph-vpapnoi-ibm-hmmiwn-node1-installer yum.repos.d]# cat ceph_stable_\$basearch.repo [ceph_stable_$basearch] baseurl = https://public.dhe.ibm.com/ibmdl/export/pub/storage/ceph/6/rhel9//$basearch gpgcheck = 1 gpgkey = https://public.dhe.ibm.com/ibmdl/export/pub/storage/ceph/RPM-GPG-KEY-IBM-CEPH name = IBM Ceph repo - $basearch priority = 2 It was noticed here that the "baseurl" contains an extra "/" just before "$basearch". It is unclear if this behavior is expected or not. Also, if this repo is removed, the ceph-common package is successfully installed when done manually: ------ [root@ceph-vpapnoi-ibm-hmmiwn-node2 yum.repos.d]# dnf install cephadm-ansible Updating Subscription Management repositories. This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions. IBM Ceph repo - x86_64 772 B/s | 196 B 00:00 Errors during downloading metadata for repository 'ceph_stable_x86_64': - Status code: 404 for https://public.dhe.ibm.com/ibmdl/export/pub/storage/ceph/6/rhel9//x86_64/repodata/repomd.xml (IP: 170.225.126.18) Error: Failed to download metadata for repo 'ceph_stable_x86_64': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried [root@ceph-vpapnoi-ibm-hmmiwn-node2 yum.repos.d]# rm -rfv ceph_stable_\$basearch.repo removed 'ceph_stable_$basearch.repo' [root@ceph-vpapnoi-ibm-hmmiwn-node2 yum.repos.d]# dnf install ceph-common Updating Subscription Management repositories. This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions. Last metadata expiration check: 0:52:53 ago on Wed 21 Jun 2023 10:01:26 AM EDT. Dependencies resolved. ============================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================ Installing: ceph-common x86_64 2:17.2.6-70.el9cp IBM-CEPH-6.1-20230614.ci.0 23 M Installing dependencies: daxctl-libs x86_64 71.1-8.el9 rhel-9-for-x86_64-baseos-rpms 44 k gperftools-libs x86_64 2.9.1-2.el9cp ibm-storage-ceph-5 311 k libbabeltrace x86_64 1.5.8-10.el9 rhel-9-for-x86_64-appstream-rpms 194 k libcephfs2 x86_64 2:17.2.6-70.el9cp IBM-CEPH-6.1-20230614.ci.0 655 k liboath x86_64 2.6.7-2.el9cp ibm-storage-ceph-5 50 k libpmem x86_64 1.12.1-1.el9 rhel-9-for-x86_64-appstream-rpms 115 k libpmemobj x86_64 1.12.1-1.el9 rhel-9-for-x86_64-appstream-rpms 163 k librabbitmq x86_64 0.11.0-5.el9 rhel-9-for-x86_64-appstream-rpms 48 k librados2 x86_64 2:17.2.6-70.el9cp IBM-CEPH-6.1-20230614.ci.0 3.1 M libradosstriper1 x86_64 2:17.2.6-70.el9cp IBM-CEPH-6.1-20230614.ci.0 479 k librbd1 x86_64 2:17.2.6-70.el9cp IBM-CEPH-6.1-20230614.ci.0 3.0 M librdkafka x86_64 1.6.1-102.el9 rhel-9-for-x86_64-appstream-rpms 666 k librdmacm x86_64 44.0-2.el9 rhel-9-for-x86_64-baseos-rpms 74 k librgw2 x86_64 2:17.2.6-70.el9cp IBM-CEPH-6.1-20230614.ci.0 5.0 M libunwind x86_64 1.5.0-1.el9cp ibm-storage-ceph-5 67 k lttng-ust x86_64 2.12.0-6.el9 rhel-9-for-x86_64-appstream-rpms 296 k --- snip ---