Description of problem: During the upgrade from RHCS 6.1z1 to RHCS 6.1z2, the cephadm-preflight playbook is failing on the task to disable older repositories. ~~~ TASK [disable older rhceph repositories if any] ********************************************************************************************************************************************************************************************** Monday 20 November 2023 07:05:28 +0530 (0:00:59.891) 0:01:04.482 ******* fatal: [rhcs5node1]: FAILED! => changed=false msg: rhceph-4-mon-for-rhel-9-x86_64-rpms is not a valid repository ID results: - rhceph-4-mon-for-rhel-9-x86_64-rpms is not a valid repository ID fatal: [rhcs5node2]: FAILED! => changed=false msg: rhceph-4-tools-for-rhel-9-x86_64-rpms is not a valid repository ID results: - rhceph-4-tools-for-rhel-9-x86_64-rpms is not a valid repository ID fatal: [rhcs5node3]: FAILED! => changed=false msg: rhceph-4-mon-for-rhel-9-x86_64-rpms is not a valid repository ID results: - rhceph-4-mon-for-rhel-9-x86_64-rpms is not a valid repository ID ~~~ Version-Release number of selected component (if applicable): ansible-core-2.13.3-2.el9_1.x86_64 ansible-collection-community-general-4.0.0-1.1.el9ost.noarch ansible-collection-ansible-posix-1.2.0-1.3.el9ost.noarch cephadm-17.2.6-148.el9cp.noarch cephadm-ansible-3.0.0-1.el9cp.noarch RHEL 9.1 How reproducible: It is happening all the time Steps to Reproduce: 1. Update the cephadm and cephadm-ansible to the above mentioned version 2. Then run the preflight playbook as per the upgrade steps 3. It fails on the task to disable older RHCS repositories as there is no RHCS 4 repo for RHEL 9 Actual results: The playbook fails to execute Expected results: The playbook execution should be completed successfully. Additional info: If I try to disable the variable related to the RHCS 4 repositories, then it goes through the disabling older repo task. ~~~ [root@rhcs5node1 cephadm-ansible]# grep -A 3 repos_to_disable cephadm-preflight.yml repos_to_disable: # - rhceph-4-tools-for-rhel-{{ ansible_facts['distribution_major_version'] }}-{{ ansible_facts['architecture'] }}-rpms # - rhceph-4-mon-for-rhel-{{ ansible_facts['distribution_major_version'] }}-{{ ansible_facts['architecture'] }}-rpms # - rhceph-4-osd-for-rhel-{{ ansible_facts['distribution_major_version'] }}-{{ ansible_facts['architecture'] }}-rpms ~~~ The task **disable older rhceph repositories if any** completes successfully. ~~~ TASK [disable older rhceph repositories if any] ********************************************************************************************************************************************************************************************** Monday 20 November 2023 07:27:24 +0530 (0:00:26.002) 0:00:26.422 ******* ok: [rhcs5node1] ok: [rhcs5node3] ok: [rhcs5node2] ~~~ But this time it fails in the task to install ceph-common package due to the wrong repo ~~~ TASK [install ceph-common on rhel] *********************************************************************************************************************************************************************************************************** Monday 20 November 2023 07:27:51 +0530 (0:00:00.125) 0:00:53.540 ******* FAILED - RETRYING: [rhcs5node1]: install ceph-common on rhel (3 retries left). FAILED - RETRYING: [rhcs5node3]: install ceph-common on rhel (3 retries left). FAILED - RETRYING: [rhcs5node2]: install ceph-common on rhel (3 retries left). FAILED - RETRYING: [rhcs5node1]: install ceph-common on rhel (2 retries left). FAILED - RETRYING: [rhcs5node3]: install ceph-common on rhel (2 retries left). FAILED - RETRYING: [rhcs5node2]: install ceph-common on rhel (2 retries left). FAILED - RETRYING: [rhcs5node2]: install ceph-common on rhel (1 retries left). FAILED - RETRYING: [rhcs5node3]: install ceph-common on rhel (1 retries left). FAILED - RETRYING: [rhcs5node1]: install ceph-common on rhel (1 retries left). fatal: [rhcs5node2]: FAILED! => changed=false attempts: 3 cmd: - dnf - install - --allowerasing - --assumeyes - ceph-common delta: '0:00:05.770649' end: '2023-11-20 07:28:34.186924' msg: non-zero return code rc: 1 start: '2023-11-20 07:28:28.416275' stderr: |- Errors during downloading metadata for repository 'rhceph-7-tools-for-rhel-9-x86_64-rpms': - Status code: 404 for https://cdn.redhat.com/content/dist/layered/rhel9/x86_64/rhceph-tools/7/os/repodata/repomd.xml (IP: 23.58.8.251) Error: Failed to download metadata for repo 'rhceph-7-tools-for-rhel-9-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried stderr_lines: <omitted> stdout: |- Updating Subscription Management repositories. Red Hat Ceph Storage Tools 7 for RHEL 9 x86_64 4.6 B/s | 10 B 00:02 stdout_lines: <omitted> fatal: [rhcs5node3]: FAILED! => changed=false attempts: 3 cmd: - dnf - install - --allowerasing - --assumeyes - ceph-common delta: '0:00:06.616174' end: '2023-11-20 07:28:35.942302' msg: non-zero return code rc: 1 start: '2023-11-20 07:28:29.326128' stderr: |- Errors during downloading metadata for repository 'rhceph-7-tools-for-rhel-9-x86_64-rpms': - Status code: 404 for https://cdn.redhat.com/content/dist/layered/rhel9/x86_64/rhceph-tools/7/os/repodata/repomd.xml (IP: 23.58.8.251) Error: Failed to download metadata for repo 'rhceph-7-tools-for-rhel-9-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried stderr_lines: <omitted> stdout: |- Updating Subscription Management repositories. Red Hat Ceph Storage Tools 7 for RHEL 9 x86_64 6.8 B/s | 10 B 00:01 stdout_lines: <omitted> fatal: [rhcs5node1]: FAILED! => changed=false attempts: 3 cmd: - dnf - install - --allowerasing - --assumeyes - ceph-common delta: '0:00:06.803554' end: '2023-11-20 07:28:36.304528' msg: non-zero return code rc: 1 start: '2023-11-20 07:28:29.500974' stderr: |- Errors during downloading metadata for repository 'rhceph-7-tools-for-rhel-9-x86_64-rpms': - Status code: 404 for https://cdn.redhat.com/content/dist/layered/rhel9/x86_64/rhceph-tools/7/os/repodata/repomd.xml (IP: 23.58.8.251) Error: Failed to download metadata for repo 'rhceph-7-tools-for-rhel-9-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried stderr_lines: <omitted> stdout: |- Updating Subscription Management repositories. Red Hat Ceph Storage Tools 7 for RHEL 9 x86_64 4.6 B/s | 10 B 00:02 stdout_lines: <omitted> ~~~ So looks like in the enable repo task instead of RHCS 6 repository, RHCS 7 repos are added. RHCS 7 repo is not yet live and hence installing the ceph-common package from the repo fails. ~~~ [root@rhcs5node1 cephadm-ansible]# dnf repolist Updating Subscription Management repositories. repo id repo name rhceph-5-tools-for-rhel-9-x86_64-rpms Red Hat Ceph Storage Tools 5 for RHEL 9 x86_64 (RPMs) rhceph-6-tools-for-rhel-9-x86_64-rpms Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) rhceph-7-tools-for-rhel-9-x86_64-rpms Red Hat Ceph Storage Tools 7 for RHEL 9 x86_64 (RPMs) rhel-9-for-x86_64-appstream-rpms Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) rhel-9-for-x86_64-baseos-rpms Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) ~~~ While checking the playbook, found that **ceph_rhcs_version** is getting tagged as 7 wrongly instead of 6. ~~~ [root@rhcs5node1 cephadm-ansible]# cd ceph_defaults/ [root@rhcs5node1 ceph_defaults]# ll total 4 drwxr-xr-x. 2 root root 22 Nov 20 07:03 defaults drwxr-xr-x. 2 root root 22 Nov 20 07:03 meta -rw-r--r--. 1 root root 100 Jul 12 17:54 README.md [root@rhcs5node1 ceph_defaults]# cd defaults/ [root@rhcs5node1 defaults]# ll total 4 -rw-r--r--. 1 root root 764 Jul 12 17:54 main.yml [root@rhcs5node1 defaults]# grep -i ceph_rhcs_version main.yml ceph_rhcs_version: 7 ~~~ So changed the default value from 7 to 6, and disabled RHCS 7 repo which was wrongly enabled by the playbook. After this the playbook executed successfully. ~~~ [root@rhcs5node1 cephadm-ansible]# grep version ceph_defaults/defaults/main.yml ceph_rhcs_version: 6 ceph_ibm_version: 6 [root@rhcs5node1 cephadm-ansible]# dnf config-manager --save --disable rhceph-7-tools-for-rhel-9-x86_64-rpms Updating Subscription Management repositories. [root@rhcs5node1 cephadm-ansible]# ssh rhcs5node2 "dnf config-manager --save --disable rhceph-7-tools-for-rhel-9-x86_64-rpms" Updating Subscription Management repositories. [root@rhcs5node1 cephadm-ansible]# ssh rhcs5node3 "dnf config-manager --save --disable rhceph-7-tools-for-rhel-9-x86_64-rpms" Updating Subscription Management repositories. ~~~ The playbook execution: ~~~ PLAY RECAP *********************************************************************************************************************************************************************************************************************************** localhost : ok=0 changed=0 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0 rhcs5node1 : ok=5 changed=2 unreachable=0 failed=0 skipped=23 rescued=0 ignored=0 rhcs5node2 : ok=5 changed=2 unreachable=0 failed=0 skipped=23 rescued=0 ignored=0 rhcs5node3 : ok=5 changed=2 unreachable=0 failed=0 skipped=23 rescued=0 ignored=0 ~~~