This would also affect 16.2 and 16.2.1 +++ This bug was initially created as a clone of Bug #2026654 +++ Description of problem: When updating the overcloud, the undercloud registry fails to provide the correct images, if the following conditions are met: 1. Used containers-prepare-parameter.yaml with 16.1.x tag (with x < 6, as of this writing) to perform the initial deployment of RHOSP 2. Used containers-prepare-parameter.yaml with 16.1 tag when preparing the udpate This causes the MultiView handler on apache to always return the older image, even if the 16.1 tag is specified: [root@controller1-161 ~]# podman images |grep aodh director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator 16.1 d6480804c4d2 8 months ago 743 MB director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator 16.1.4 d6480804c4d2 8 months ago 743 MB How reproducible: Always reproducible by pulling images from CDN to prepare overcloud upgrade, if tag 16.1.4 is used first, and then 16.1. Steps to Reproduce: 1. Deploy undercloud and overcloud using tag 16.1.4 on the file containers-prepare-parameter.yaml $ egrep -B1 tag_from_label containers-prepare-parameter.yaml tag: '16.1.4' tag_from_label: '{version}-{release}' 2. Change tag from 16.1.4 to 16.1, in preparation for update, and as documented on [1] $ sed -i 's/16.1.4/16.1/' containers-prepare-parameter.yaml $ egrep -B1 tag_from_label containers-prepare-parameter.yaml tag: '16.1' tag_from_label: '{version}-{release}' 3. Update undercloud image registry $ sudo openstack tripleo container image prepare -e containers-prepare-parameter.yaml 4. Test for a given image, that the version with tag 16.1 has been populated on the undercloud registry (along with the older one): [stack@director ~]$ cat /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/tags/list | jq . { "name": "rhosp-rhel8/openstack-aodh-listener", "tags": [ "16.1.4", "16.1" ] } [stack@director ~]$ ll /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/*.type-map -rw-r--r--. 1 root root 169 Nov 25 11:01 /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4.type-map -rw-r--r--. 1 root root 167 Nov 25 11:40 /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.type-map [stack@director ~]$ 5. On one of the controllers, check for the current version of the same image: [root@controller1-161 ~]# podman images |grep aodh director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator 16.1.4 d6480804c4d2 8 months ago 743 MB [root@controller1-161 ~]# 6. Pull the image using the tag 16.1: [root@controller1-161 ~]# podman pull director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator:16.1 Trying to pull director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator:16.1... Getting image source signatures Copying blob ea462113cc1f skipped: already exists Copying blob fe9c8cc943e3 skipped: already exists Copying blob 4422ea938fce skipped: already exists Copying blob 00f18994d663 skipped: already exists Copying blob d8ec3e673e1e skipped: already exists Copying blob 505209649de3 skipped: already exists Copying config d6480804c4 done Writing manifest to image destination Storing signatures d6480804c4d2cb600479b46f92f728db804410d68c264566dd5d3859a21bd750 [root@controller1-161 ~]# Actual results: As a result, the controller has pulled the same image than already had, and added the tag 16.1 to it: [root@controller1-161 ~]# podman images |grep aodh director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator 16.1 d6480804c4d2 8 months ago 743 MB director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator 16.1.4 d6480804c4d2 8 months ago 743 MB [root@controller1-161 ~]# When using curl to check the images manifest, they both show the same: [root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1 | jq .config.digest "sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273" [root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4 | jq .config.digest "sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273" [root@controller1-161 ~]# However, the Apache type-map file correctly shows different digest: [root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.type-map | jq .config.digest "sha256:ea548ddc17674629c97f7472cb5fe62f5495208bcf2f27d5918bd9a8e7c9833f" [root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4.type-map | jq .config.digest "sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273" [root@controller1-161 ~]# Therefore, the type-map files have the correct content, but Apache is always using the older one. Expected results: Apache type-map should correctly map the GET request of */16.1, to the type-map file */16.1.type-map, instead of matching 16.1.4.type-map, so that podman can pull the right image. Additional info: [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/keeping_red_hat_openstack_platform_updated/index#updating-your-container-image-preparation-file_keeping-updated
Is it to early to ask on which z stream is this fix going to be included?
Procedure used was: deploy undercloud with puddle that has fix openstack tripleo container image prepare with a tag of 16.2.1 podman pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator openstack tripleo container image prepare with a tag of 16.2 podman pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator Saw that 16.2.1 and 16.2 ids were different: [stack@undercloud-0 ~]$ sudo podman images | grep aodh undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator 16.2 2548ab62702d 3 hours ago 700 MB undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator 16.2.1 ce85155e7718 2 months ago 701 MB [stack@undercloud-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1001