Description of problem: overcloud deployment failing with the error below 2022-11-07 18:10:43.930573 | 246e96c8-fa00-39d9-7140-00000000075f | TASK | enable tripleo-ip6tables service 2022-11-07 18:10:44.299753 | 246e96c8-fa00-39d9-7140-00000000075f | CHANGED | enable tripleo-ip6tables service | compute-0 2022-11-07 18:10:44.304012 | 246e96c8-fa00-39d9-7140-00000000075f | TIMING | enable tripleo-ip6tables service | compute-0 | 0:00:33.490664 | 0.46s 2022-11-07 18:10:44.416419 | 246e96c8-fa00-39d9-7140-00000000003a | SKIPPED | include_tasks | compute-0 2022-11-07 18:10:44.416926 | 246e96c8-fa00-39d9-7140-00000000003a | TIMING | include_tasks | compute-0 | 0:00:33.603582 | 0.05s 2022-11-07 18:10:44.418476 | 246e96c8-fa00-39d9-7140-00000000075f | CHANGED | enable tripleo-ip6tables service | compute-1 2022-11-07 18:10:44.422073 | 246e96c8-fa00-39d9-7140-00000000075f | TIMING | enable tripleo-ip6tables service | compute-1 | 0:00:33.608733 | 0.49s 2022-11-07 18:10:44.546095 | 246e96c8-fa00-39d9-7140-00000000003b | SKIPPED | include_tasks | compute-0 2022-11-07 18:10:44.546674 | 246e96c8-fa00-39d9-7140-00000000003b | TIMING | include_tasks | compute-0 | 0:00:33.733328 | 0.07s 2022-11-07 18:10:44.547046 | 246e96c8-fa00-39d9-7140-00000000003a | SKIPPED | include_tasks | compute-1 2022-11-07 18:10:44.547297 | 246e96c8-fa00-39d9-7140-00000000003a | TIMING | include_tasks | compute-1 | 0:00:33.733958 | 0.06s 2022-11-07 18:10:44.669172 | 246e96c8-fa00-39d9-7140-00000000003c | SKIPPED | include_tasks | compute-0 2022-11-07 18:10:44.669778 | 246e96c8-fa00-39d9-7140-00000000003c | TIMING | include_tasks | compute-0 | 0:00:33.856414 | 0.07s 2022-11-07 18:10:44.670265 | 246e96c8-fa00-39d9-7140-00000000003b | SKIPPED | include_tasks | compute-1 2022-11-07 18:10:44.670566 | 246e96c8-fa00-39d9-7140-00000000003b | TIMING | include_tasks | compute-1 | 0:00:33.857209 | 0.06s 2022-11-07 18:10:44.795830 | 246e96c8-fa00-39d9-7140-00000000003c | SKIPPED | include_tasks | compute-1 2022-11-07 18:10:44.796303 | 246e96c8-fa00-39d9-7140-00000000003c | TIMING | include_tasks | compute-1 | 0:00:33.982959 | 0.06s 2022-11-07 18:10:44.853744 | 246e96c8-fa00-39d9-7140-000000000033 | RETRY | Wait for cloud-init to finish, if enabled | controller-2 | 47 retries left 2022-11-07 18:14:48.573849 | 246e96c8-fa00-39d9-7140-000000000033 | FATAL | Wait for cloud-init to finish, if enabled | controller-2 | error={"ansible_facts": {"cloud_init_data_facts": {"status": {"v1": {"datasource": "DataSourceConfigDrive [net,ver=2][source=/dev/sda1]", "init": {"errors": [], "finished": 1667844533.4307442, "start": 1667844531.9871407}, "init-local": {"errors": [], "finished": 1667844524.9883764, "start": 1667844524.6055927}, "modules-config": {"errors": [], "finished": 1667844534.021143, "start": 1667844533.874322}, "modules-final": {"errors": [], "finished": null, "start": 1667844534.4129207}, "modules-init": {"errors": [], "finished": null, "start": null}, "stage": "modules-final"}}}}, "attempts": 50, "changed": false, "cloud_init_data_facts": {"status": {"v1": {"datasource": "DataSourceConfigDrive [net,ver=2][source=/dev/sda1]", "init": {"errors": [], "finished": 1667844533.4307442, "start": 1667844531.9871407}, "init-local": {"errors": [], "finished": 1667844524.9883764, "start": 1667844524.6055927}, "modules-config": {"errors": [], "finished": 1667844534.021143, "start": 1667844533.874322}, "modules-final": {"errors": [], "finished": null, "start": 1667844534.4129207}, "modules-init": {"errors": [], "finished": null, "start": null}, "stage": "modules-final"}}}} The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden Overcloud configuration failed. 2022-11-07 18:14:48.575466 | 246e96c8-fa00-39d9-7140-000000000033 | TIMING | Wait for cloud-init to finish, if enabled | controller-2 | 0:04:37.762120 | 259.58s log: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/138/sealusa39.mobius.lab.eng.rdu2.redhat.com/home/stack/overcloud_install.log.gz Version-Release number of selected component (if applicable): 16.2 How reproducible: 100% Steps to Reproduce: 1. Deploy openstack 16.2 in jenkins https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/enterprise/view/scenario/job/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/ 2. At the stage "overcloud", deployment is failing console log: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/enterprise/view/scenario/job/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/138/consoleFull error log: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/138/sealusa39.mobius.lab.eng.rdu2.redhat.com/home/stack/overcloud_install.log.gz Actual results: overcloud deployment is failing Expected results: pass Additional info:
This does not look like a product but, it looks like package installation is attempted without any repositories configured. cloud-init fails during package installation: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/138/controller-2/var/log/cloud-init-output.log.gz Cloud-init v. 20.3-10.el8_4.11 running 'modules:config' at Mon, 07 Nov 2022 18:08:53 +0000. Up 23.30 seconds. Updating Subscription Management repositories. Unable to read consumer identity This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. There are no enabled repositories in "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d". Updating Subscription Management repositories. Unable to read consumer identity This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Error: There are no enabled repositories in "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d". Cloud-init v. 20.3-10.el8_4.11 running 'modules:final' at Mon, 07 Nov 2022 18:08:54 +0000. Up 23.84 seconds. 2022-11-07 18:08:55,576 - util.py[WARNING]: Failed to install packages: ['python-simplejson', 'ipa-client', 'ipa-admintools', 'openldap-clients', 'hostname'] 2022-11-07 18:08:55,579 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one 2022-11-07 18:08:55,579 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/138/controller-2/var/log/cloud-init.log.gz 2022-11-07 18:08:54,494 - rhel.py[DEBUG]: Using DNF for package management 2022-11-07 18:08:54,494 - subp.py[DEBUG]: Running command ['dnf', '-y', 'makecache'] with allowed return codes [0] (shell=False, capture=False) 2022-11-07 18:08:55,095 - rhel.py[DEBUG]: Using DNF for package management 2022-11-07 18:08:55,095 - subp.py[DEBUG]: Running command ['dnf', '-y', 'install', 'python-simplejson', 'ipa-client', 'ipa-admintools', 'openldap-clients', 'hostname'] with allowed return codes [0] (shell=False, capture=False) 2022-11-07 18:08:55,576 - util.py[WARNING]: Failed to install packages: ['python-simplejson', 'ipa-client', 'ipa-admintools', 'openldap-clients', 'hostname'] 2022-11-07 18:08:55,577 - util.py[DEBUG]: Failed to install packages: ['python-simplejson', 'ipa-client', 'ipa-admintools', 'openldap-clients', 'hostname'] Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 100, in handle cloud.distro.install_packages(pkglist) File "/usr/lib/python3.6/site-packages/cloudinit/distros/rhel.py", line 63, in install_packages self.package_command('install', pkgs=pkglist) File "/usr/lib/python3.6/site-packages/cloudinit/distros/rhel.py", line 177, in package_command subp.subp(cmd, capture=False) File "/usr/lib/python3.6/site-packages/cloudinit/subp.py", line 293, in subp cmd=args) cloudinit.subp.ProcessExecutionError: Unexpected error while running command. Command: ['dnf', '-y', 'install', 'python-simplejson', 'ipa-client', 'ipa-admintools', 'openldap-clients', 'hostname'] Exit code: 1
Moving off DFG:HardProv, since installing packages in a cloud-init workflow is outside the remit of baremetal provisioning.