Description of problem: periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy with below traceback : 2022-12-14 04:56:29.712233 | primary | TASK [Run container image prepare for compute deploy] ************************** 2022-12-14 04:56:29.712434 | primary | Wednesday 14 December 2022 09:56:29 +0000 (0:00:02.767) 0:00:19.992 **** 2022-12-14 05:01:24.418433 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "source /home/zuul/stackrc\nsudo openstack tripleo container image prepare -e /home/zuul/containers-prepare-parameter.yaml\n", "delta": "0:04:53.205787", "end": "2022-12-14 10:01:24.121281", "msg": "non-zero return code", "rc": 1, "start": "2022-12-14 09:56:30.915494", "stderr": "Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-container-image-prepare.yaml, Run Status: failed, Return Code: 2\nException occured while running the command\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/tripleoclient/command.py\", line 32, in run\n super(Command, self).run(parsed_args)\n File \"/usr/lib/python3.9/site-packages/osc_lib/command/command.py\", line 39, in run\n return super(Command, self).run(parsed_args)\n File \"/usr/lib/python3.9/site-packages/cliff/command.py\", line 186, in run\n return_code = self.take_action(parsed_args) or 0\n File \"/usr/lib/python3.9/site-packages/tripleoclient/v1/container_image.py\", line 1092, in take_action\n oooutils.run_ansible_playbook(\n File \"/usr/lib/python3.9/site-packages/tripleoclient/utils.py\", line 735, in run_ansible_playbook\n raise RuntimeError(err_msg)\nRuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-container-image-prepare.yaml, Run Status: failed, Return Code: 2\nAnsible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-container-image-prepare.yaml, Run Status: failed, Return Code: 2", "stderr_lines": ["Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-container-image-prepare.yaml, Run Status: failed, Return Code: 2", "Exception occured while running the command", "Traceback (most recent call last):", " File \"/usr/lib/python3.9/site-packages/tripleoclient/command.py\", line 32, in run", " super(Command, self).run(parsed_args)", " File \"/usr/lib/python3.9/site-packages/osc_lib/command/command.py\", line 39, in run", " return super(Command, self).run(parsed_args)", " File \"/usr/lib/python3.9/site-packages/cliff/command.py\", line 186, in run", " return_code = self.take_action(parsed_args) or 0", " File \"/usr/lib/python3.9/site-packages/tripleoclient/v1/container_image.py\", line 1092, in take_action", " oooutils.run_ansible_playbook(", " File \"/usr/lib/python3.9/site-packages/tripleoclient/utils.py\", line 735, in run_ansible_playbook", " raise RuntimeError(err_msg)", "RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-container-image-prepare.yaml, Run Status: failed, Return Code: 2", "Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-container-image-prepare.yaml, Run Status: failed, Return Code: 2"], "stdout": "[DEPRECATION WARNING]: ANSIBLE_CALLBACK_WHITELIST option, normalizing names to \r\nnew standard, use ANSIBLE_CALLBACKS_ENABLED instead. This feature will be \r\nremoved from ansible-core in version 2.15. Deprecation warnings can be disabled\r\n by setting deprecation_warnings=False in ansible.cfg.\r\n\r\nPLAY [Container Image Prepare] *************************************************\n2022-12-14 09:56:33.889983 | fa163e7c-c0fc-2f7e-9ce5-000000000008 | TASK | Fetch roles_data\n2022-12-14 09:56:34.290544 | fa163e7c-c0fc-2f7e-9ce5-000000000008 | OK | Fetch roles_data | localhost\n2022-12-14 09:56:34.293012 | fa163e7c-c0fc-2f7e-9ce5-000000000008 | TIMING | Fetch roles_data | localhost | 0:00:00.450000 | 0.40s\n2022-12-14 09:56:34.300649 | fa163e7c-c0fc-2f7e-9ce5-000000000009 | TASK | Set fact for log file\n2022-12-14 09:56:34.337012 | fa163e7c-c0fc-2f7e-9ce5-000000000009 | OK | Set fact for log file | localhost\n2022-12-14 09:56:34.338127 | fa163e7c-c0fc-2f7e-9ce5-000000000009 | TIMING | Set fact for log file | localhost | 0:00:00.495117 | 0.04s\n2022-12-14 09:56:34.344478 | fa163e7c-c0fc-2f7e-9ce5-00000000000a | TASK | Get all files in directories\n2022-12-14 09:56:34.827222 | | WARNING | Skipped '/root/.tripleo/environments' path due to this access issue: '/root/.tripleo/environments' is not a directory\n2022-12-14 09:56:34.828369 | fa163e7c-c0fc-2f7e-9ce5-00000000000a | OK | Get all files in directories | localhost\n2022-12-14 09:56:34.829076 | fa163e7c-c0fc-2f7e-9ce5-00000000000a | TIMING | Get all files in directories | localhost | 0:00:00.986074 | 0.48s\n2022-12-14 09:56:34.833704 | fa163e7c-c0fc-2f7e-9ce5-00000000000b | TASK | Build environment_files\n2022-12-14 09:56:34.869537 | fa163e7c-c0fc-2f7e-9ce5-00000000000b | OK | Build environment_files | localhost\n2022-12-14 09:56:34.870786 | fa163e7c-c0fc-2f7e-9ce5-00000000000b | TIMING | Build environment_files | localhost | 0:00:01.027757 | 0.04s\n2022-12-14 09:56:34.875852 | fa163e7c-c0fc-2f7e-9ce5-00000000000c | TASK | Build heat stack environment\n2022-12-14 09:56:36.198290 | fa163e7c-c0fc-2f7e-9ce5-00000000000c | CHANGED | Build heat stack environment | localhost\n2022-12-14 09:56:36.199781 | fa163e7c-c0fc-2f7e-9ce5-00000000000c | TIMING | Build heat stack environment | localhost | 0:00:02.356752 | 1.32s\n2022-12-14 09:56:36.206335 | fa163e7c-c0fc-2f7e-9ce5-00000000000d | TASK | Prepare container images, check logs in /root/container_image_prepare.log\n2022-12-14 10:01:23.201326 | fa163e7c-c0fc-2f7e-9ce5-00000000000d | FATAL | Prepare container images, check logs in /root/container_image_prepare.log | localhost | error={\"changed\": false, \"error\": \"Modifying image 192.168.24.1:8787/tripleorhos-17-1-rhel-8/openstack-nova-scheduler:2470e639dbd7f8721873871ec5be6a9e failed\", \"msg\": \"Error running container image prepare: Modifying image 192.168.24.1:8787/tripleorhos-17-1-rhel-8/openstack-nova-scheduler:2470e639dbd7f8721873871ec5be6a9e failed\", \"params\": {}, \"success\": false}\n2022-12-14 10:01:23.204008 | fa163e7c-c0fc-2f7e-9ce5-00000000000d | TIMING | Prepare container images, check logs in {{ cip_log_file }} | localhost | 0:04:49.360991 | 287.00s\n\r\nNO MORE HOSTS LEFT *************************************************************\n\r\nPLAY RECAP *********************************************************************\nlocalhost : ok=5 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 \n2022-12-14 10:01:23.210003 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n2022-12-14 10:01:23.210387 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n2022-12-14 10:01:23.210794 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:04:49.367799 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n2022-12-14 10:01:23.211173 | UUID | Info | Host | Task Name | Run Time\n2022-12-14 10:01:23.211528 | fa163e7c-c0fc-2f7e-9ce5-00000000000d | SUMMARY | localhost | Prepare container images, check logs in {{ cip_log_file }} | 287.00s\n2022-12-14 10:01:23.211937 | fa163e7c-c0fc-2f7e-9ce5-00000000000c | SUMMARY | localhost | Build heat stack environment | 1.32s\n2022-12-14 10:01:23.212395 | fa163e7c-c0fc-2f7e-9ce5-00000000000a | SUMMARY | localhost | Get all files in directories | 0.48s\n2022-12-14 10:01:23.212732 | fa163e7c-c0fc-2f7e-9ce5-000000000008 | SUMMARY | localhost | Fetch roles_data | 0.40s\n2022-12-14 10:01:23.213116 | fa163e7c-c0fc-2f7e-9ce5-00000000000b | SUMMARY | localhost | Build environment_files | 0.04s\n2022-12-14 10:01:23.213458 | fa163e7c-c0fc-2f7e-9ce5-000000000009 | SUMMARY | localhost | Set fact for log file | 0.04s\n2022-12-14 10:01:23.213798 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n2022-12-14 10:01:23.214187 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n2022-12-14 10:01:23.214535 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
*** Bug 2154197 has been marked as a duplicate of this bug. ***
In the yum_update.sh log there is a mix of el8 and el9 builds, is that expected? what that task should do?
@amoralej it is both because it is mixed OS job. For the issue in this bug, the problem is with the el8 container update i.e. running container image prepare and part of that process is running a yum update which fails for the package conflict here
(In reply to Marios Andreou from comment #6) > @amoralej it is both because it is mixed OS job. > > For the issue in this bug, the problem is with the el8 container update i.e. > running container image prepare and part of that process is running a yum > update which fails for the package conflict here It's trying to install some el9 builds. I see that tasks is mapping the host repos config /etc/yum.repos.d:/etc/yum.repos.d: https://sf.hosted.upshift.rdu2.redhat.com/logs/04/437804/3/check/periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal/5483684/logs/undercloud/etc/yum.repos.d/ So i guess it needs some logic to use only el8 repos in the el8 container update.
workaround to disable yum update on centos8 content in the mixed rhel component jobs posted there [1] [1] https://review.opendev.org/c/openstack/tripleo-ci/+/870022 Add update_containers: false for mixed rhel component jobs