Description of problem:

Update deployment of overcloud fails with error message: Image registry.redhat.io/rhosp-rhel8/openstack-aodh-api has no tag 16.2.


Version-Release number of selected component (if applicable):

Red Hat OpenStack Platform release 16.2.2 (Train)
openstack-tripleo-common-11.7.1-2.20211218004850.173edba.el8ost.noarch Thu May 12 12:57:26 2022


How reproducible:

Always


Steps to Reproduce:

1. Configure container image preparation file (default: containers-prepare-parameter.yaml) without `tag: 16.2` parameter as follows.

parameter_defaults:
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      'XXXXXXX' : 'XXXXXXX'
  ContainerImagePrepare:
  - push_destination: true
    set:
      ceph_alertmanager_image: ose-prometheus-alertmanager
      ceph_alertmanager_namespace: registry.redhat.io/openshift4
      ceph_alertmanager_tag: 4.1
      ceph_grafana_image: rhceph-4-dashboard-rhel8
      ceph_grafana_namespace: registry.redhat.io/rhceph
      ceph_grafana_tag: 4
      ceph_image: rhceph-4-rhel8
      ceph_namespace: registry.redhat.io/rhceph
      ceph_node_exporter_image: ose-prometheus-node-exporter
      ceph_node_exporter_namespace: registry.redhat.io/openshift4
      ceph_node_exporter_tag: v4.1
      ceph_prometheus_image: ose-prometheus
      ceph_prometheus_namespace: registry.redhat.io/openshift4
      ceph_prometheus_tag: 4.1
      ceph_tag: latest
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-rhel8
      neutron_driver: ovn
      rhel_containers: false
    tag_from_label: '{version}-{release}'

2. Deploy overcloud.


Actual results:

Deployment fails with following messages.

WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon! GnocchiArchivePolicy, NovaComputeSchedulerHints
WARNING: Following parameter(s) are defined but not currently used in the deployment plan. These parameters may be valid but not in use due to the service or deployment configuration. AlertManagerContainerImage, GrafanaContainerImage, NovaComputeAvailabilityZone, PrometheusContainerImage, SwiftRingGetTempurl, SwiftRingPutTempurl
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1069, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 435, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 552, in _deploy_tripleo_heat_templates
    deployment_options=deployment_options)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 571, in _try_overcloud_deploy_with_compat_yaml
    deployment_options=deployment_options)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 308, in _heat_deploy
    deployment_options=deployment_options)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 87, in deploy_and_wait
    deploy(log, clients, **workflow_input)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 69, in deploy
    % (payload['status'], wf_name))
ValueError: Unexpected status FAILED for tripleo.deployment.v1.deploy_plan
Unexpected status FAILED for tripleo.deployment.v1.deploy_plan
Removing the current plan files
Uploading new plan files
Temporary Swift GET/PUT URL parameters have successfully been updated.
Plan updated.
Processing templates in the directory /tmp/tripleoclient-l_tq6jef/tripleo-heat-templates
Deploying templates in the directory /tmp/tripleoclient-l_tq6jef/tripleo-heat-templates
Initializing overcloud plan deployment
Image registry.redhat.io/rhosp-rhel8/openstack-aodh-api has no tag 16.2.
Available tags: 16.1.3-5.1610364748, 16.1.7-10.1646285508, 16.1.6-6, 16.0-79, 16.2.1-6, 16.0-106, 16.0, 16.1.5-1.1618379098, 16.1.8-9, 16.2.0-66-source, 16.1-50, 16.2.2-16-source, 16.2.3-9-source, 16.1.8-10.1655151541-source, 16.1.8-10-source, 16.0-104, 16.1.8-7-source, 16.1.8-9.1651483883-source, 16.2.0-70-source, 16.1.7-10.1643120407-source, 16.2.1-6.1645706313-source, 16.2.2-12, 16.2.2-14-source, 16.2.1-6.1645706313, 16.1.8-10.1655151541, 16.1.6-6.1634111158, 16.0-105, 16.1.8-7, 16.1.6-6.1634111158-source, 16.1.3-6.1614770396, 16.1.3-5.1611701331-source, 16.1.3-6.1614770396-source, 16.2.1-6.1643118678-source, 16.1.7, 16.2.1-6.1643118678, 16.1-45, 16.1.7-10.1643120407, 16.1.4, 16.2.2-12-source, 16.2.3-9, 16.1.6, 16.1-51, 16.1.3-6, 16.1.7-10, 16.1.6-6.1627296110, 16.1-50-source, 16.1.8-10, 16.1.3-5, 16.2.2-16, 16.1.6-6.1627296110-source


Expected results:

deployment to succeed


Additional info:

1. I reproduced this problem in my RHOSP 16.2 test environment during both director installation and overcloud deployment.

2. RHOSP 16.1 also show the same problem which is reported as https://bugzilla.redhat.com/show_bug.cgi?id=2221198.