Description of problem: Performing a undercloud update of osp 17.0. After running the "openstack undercloud upgrade" command(the upgrade ran successfully) but it is not possible to run openstack overcloud update prepare, because openstack-heat-* are purged at the end of undercloud upgrade. Openstack-heat-* iamges are required to run ephemeral-heat. Version-Release number of selected component (if applicable): 17.0_20211207.1 Steps to Reproduce: 1. Install 17.0 undercloud 2. Install 17.0 overcloud 3. Run "openstack undercloud upgrade" 4. Run "openstack overcloud update prepare" - failure, beacuase of missing ephemeral heat images on undercloud Actual results: openstack-heat-* images are pulled as part of undercloud update first,but then purged at the end of undercloud update. (undercloud) [stack@undercloud-0 ~]$ grep -ne "Pull " undercloud_update_verbose.log 141935:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 141937:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 141941:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 141942:2022-01-18 13:34:56.133929 | 52540060-0343-fa83-a946-0000000025ae | TASK | Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-all:17.0_20211207.1 image 142149:2022-01-18 13:34:57.394655 | 52540060-0343-fa83-a946-0000000025ae | CHANGED | Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-all:17.0_20211207.1 image | undercloud-0 | result={ 142190:2022-01-18 13:34:57.396114 | 52540060-0343-fa83-a946-0000000025ae | TIMING | tripleo_container_tag : Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-all:17.0_20211207.1 image | undercloud-0 | 0:10:09.252122 | 1.26s 142295:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142297:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142301:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142302:2022-01-18 13:34:57.973617 | 52540060-0343-fa83-a946-0000000025bd | TASK | Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-engine:17.0_20211207.1 image 142381:2022-01-18 13:34:58.440765 | 52540060-0343-fa83-a946-0000000025bd | CHANGED | Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-engine:17.0_20211207.1 image | undercloud-0 | result={ 142422:2022-01-18 13:34:58.442644 | 52540060-0343-fa83-a946-0000000025bd | TIMING | tripleo_container_tag : Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-engine:17.0_20211207.1 image | undercloud-0 | 0:10:10.298659 | 0.47s 142523:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142525:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142527:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142531:host: undercloud-0, task: TASK: tripleo_container_tag : Pull {{ tripleo_container_image }} image 142532:2022-01-18 13:34:59.032328 | 52540060-0343-fa83-a946-0000000025cc | TASK | Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-api:17.0_20211207.1 image 142625:2022-01-18 13:34:59.608739 | 52540060-0343-fa83-a946-0000000025cc | CHANGED | Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-api:17.0_20211207.1 image | undercloud-0 | result={ 142666:2022-01-18 13:34:59.610130 | 52540060-0343-fa83-a946-0000000025cc | TIMING | tripleo_container_tag : Pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-api:17.0_20211207.1 image | undercloud-0 | 0:10:11.466125 | 0.58s (undercloud) [stack@undercloud-0 ~]$ grep -ne "Clean podman images " undercloud_update_verbose.log [[0;33m2022-01-18 13:35:59.743993 | 52540060-0343-a074-f6c0-00000000011e | CHANGED | Clean podman images | undercloud-0 | result={^[[0m^M ^[[0;33m "changed": true,^[[0m^M ^[[0;33m "cmd": [^[[0m^M ^[[0;33m "podman",^[[0m^M ^[[0;33m "image",^[[0m^M ^[[0;33m "prune",^[[0m^M ^[[0;33m "-a",^[[0m^M ^[[0;33m "-f"^[[0m^M ^[[0;33m ],^[[0m^M ^[[0;33m "delta": "0:00:00.853988",^[[0m^M ^[[0;33m "end": "2022-01-18 13:35:59.713919",^[[0m^M ^[[0;33m "invocation": {^[[0m^M ^[[0;33m "module_args": {^[[0m^M ^[[0;33m "_raw_params": "podman image prune -a -f",^[[0m^M ^[[0;33m "_uses_shell": false,^[[0m^M ^[[0;33m "argv": null,^[[0m^M ^[[0;33m "chdir": null,^[[0m^M ^[[0;33m "creates": null,^[[0m^M ^[[0;33m "executable": null,^[[0m^M ^[[0;33m "removes": null,^[[0m^M ^[[0;33m "stdin": null,^[[0m^M ^[[0;33m "stdin_add_newline": true,^[[0m^M ^[[0;33m "strip_empty_ends": true,^[[0m^M ^[[0;33m "warn": true^[[0m^M ^[[0;33m }^[[0m^M ^[[0;33m },^[[0m^M ^[[0;33m "rc": 0,^[[0m^M ^[[0;33m "start": "2022-01-18 13:35:58.859931",^[[0m^M ^[[0;33m "stderr": "",^[[0m^M ^[[0;33m "stderr_lines": [],^[[0m^M ^[[0;33m "stdout": "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-engine:17.0_20211207.1\nundercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-api:17.0_20211207.1\nundercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-all:17.0_20211207.1",^[[0m^M ^[[0;33m "stdout_lines": [^[[0m^M ^[[0;33m "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-engine:17.0_20211207.1",^[[0m^M ^[[0;33m "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-api:17.0_20211207.1",^[[0m^M ^[[0;33m "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-heat-all:17.0_20211207.1"^[[0m^M ^[[0;33m ]^[[0m^M ^[[0;33m}^[[0m 154228:2022-01-18 13:35:59.743993 | 52540060-0343-a074-f6c0-00000000011e | CHANGED | Clean podman images | undercloud-0 | result={ 154265:2022-01-18 13:35:59.745036 | 52540060-0343-a074-f6c0-00000000011e | TIMING | tripleo_podman : Clean podman images | undercloud-0 | 0:00:04.219569 | 1.30s 154380:2022-01-18 13:36:00.219656 | 52540060-0343-a074-f6c0-00000000011e | SUMMARY | undercloud-0 | tripleo_podman : Clean podman images | 1.30s Expected results: we should not run "Clean podman images" for openstack-heat-* images, because they are required for overcloud update. Additional info:
Can you provide the actual error when you run "openstack overcloud update prepare"? We need to see the error and any log to diagnose the issue.
An error from overcloud update preprare I got: 2022-01-28 18:48:38 | 2022-01-28 18:48:38.745 48034 INFO tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Using ephemeral heat for stack operationESC[00m 2022-01-28 18:48:39 | 2022-01-28 18:48:39.472 48034 INFO tripleoclient.heat_launcher [-] Skipping container image pull.ESC[00m 2022-01-28 18:48:39 | 2022-01-28 18:48:39.626 48034 INFO tripleoclient.heat_launcher [-] Checking that database is upESC[00m 2022-01-28 18:48:39 | 2022-01-28 18:48:39.961 48034 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is upESC[00m 2022-01-28 18:48:42 | Trying to pull localhost/tripleo/openstack-heat-api:ephemeral... 2022-01-28 18:48:42 | time="2022-01-28T18:48:42Z" level=warning msg="failed, retrying in 1s ... (1/3). Error: Error initializing source docker://localhost/tripleo/openstack-heat-api:ephemeral: error pinging docker registry localhost: Get \"https://localhost/v2/\": dial tcp [::1]:443: connect: connection refused" 2022-01-28 18:48:43 | time="2022-01-28T18:48:43Z" level=warning msg="failed, retrying in 2s ... (2/3). Error: Error initializing source docker://localhost/tripleo/openstack-heat-api:ephemeral: error pinging docker registry localhost: Get \"https://localhost/v2/\": dial tcp [::1]:443: connect: connection refused" 2022-01-28 18:48:45 | time="2022-01-28T18:48:45Z" level=warning msg="failed, retrying in 4s ... (3/3). Error: Error initializing source docker://localhost/tripleo/openstack-heat-api:ephemeral: error pinging docker registry localhost: Get \"https://localhost/v2/\": dial tcp [::1]:443: connect: connection refused" 2022-01-28 18:48:49 | Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 2022-01-28 18:48:49 | Error: Error initializing source docker://localhost/tripleo/openstack-heat-api:ephemeral: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Exception occured while running the command: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 125. 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare Traceback (most recent call last): 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare super(Command, self).run(parsed_args) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare return super(Command, self).run(parsed_args) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/cliff/command.py", line 186, in run 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare return_code = self.take_action(parsed_args) or 0 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_update.py", line 78, in take_action 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare super(UpdatePrepare, self).take_action(parsed_args) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1282, in take_action 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare self.setup_ephemeral_heat(parsed_args) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 748, in setup_ephemeral_heat 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare self.orchestration_client = utils.launch_heat(self.heat_launcher) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 2719, in launch_heat 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare launcher.heat_db_sync(restore_db) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python3.6/site-packages/tripleoclient/heat_launcher.py", line 543, in heat_db_sync 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare subprocess.check_call(cmd) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare raise CalledProcessError(retcode, cmd) 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 125. 2022-01-28 18:48:49 | 2022-01-28 18:48:49.066 48034 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare ESC[00m 2022-01-28 18:48:49 | 2022-01-28 18:48:49.080 48034 ERROR openstack [-] Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 125.: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:/home/stack/overcloud-deploy/qe-Cloud-0/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 125.ESC[00m 2022-01-28 18:48:49 | 2022-01-28 18:48:49.081 48034 INFO osc_lib.shell [-] END return value: 1ESC[00m A full command of overcloud update prepare I used: openstack overcloud update prepare --yes \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack qe-Cloud-0 \ -e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \ -e /home/stack/templates/overcloud-vip-deployed.yaml \ -e /home/stack/templates/overcloud-networks-deployed.yaml \ -e /home/stack/templates/overcloud-baremetal-deployed.yaml \ -e /home/stack/virt/config_lvm.yaml \ -e /home/stack/virt/network/network-environment_v2.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e /home/stack/cli_opts_params.yaml \ -e /home/stack/skip_rhel_release.yaml \
Temporary workaround for the time being is to pull and tag the image. This example uses quay, but you wouldn't use quay for downstream, use whatever the image source is for osp17: for i in all api engine; do podman pull quay.io/tripleomaster/openstack-heat-${i}:current-tripleo ; done for i in all api engine; do podman tag quay.io/tripleomaster/openstack-heat-${i}:current-tripleo localhost/tripleo/openstack-heat-${i}:ephemeral; done
patch: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827539
wallaby patch: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827746
wallaby patch merged upstream.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543