Bug 2028962

This would also affect 16.2 and 16.2.1

+++ This bug was initially created as a clone of Bug #2026654 +++

Description of problem:
When updating the overcloud, the undercloud registry fails to provide the correct images, if the following conditions are met:

1. Used containers-prepare-parameter.yaml with 16.1.x tag (with x < 6, as of this writing) to perform the initial deployment of RHOSP
2. Used containers-prepare-parameter.yaml with 16.1 tag when preparing the udpate

This causes the MultiView handler on apache to always return the older image, even if the 16.1 tag is specified:

[root@controller1-161 ~]# podman images |grep aodh
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1         d6480804c4d2   8 months ago   743 MB
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1.4       d6480804c4d2   8 months ago   743 MB


How reproducible:
Always reproducible by pulling images from CDN to prepare overcloud upgrade, if tag 16.1.4 is used first, and then 16.1.

Steps to Reproduce:
1. Deploy undercloud and overcloud using tag 16.1.4 on the file containers-prepare-parameter.yaml

$ egrep -B1 tag_from_label containers-prepare-parameter.yaml
      tag: '16.1.4'
    tag_from_label: '{version}-{release}'

2. Change tag from 16.1.4 to 16.1, in preparation for update, and as documented on [1]

$ sed -i 's/16.1.4/16.1/' containers-prepare-parameter.yaml
$ egrep -B1 tag_from_label containers-prepare-parameter.yaml
      tag: '16.1'
    tag_from_label: '{version}-{release}'

3. Update undercloud image registry

$ sudo openstack tripleo container image prepare -e containers-prepare-parameter.yaml 

4. Test for a given image, that the version with tag 16.1 has been populated on the undercloud registry (along with the older one):

[stack@director ~]$ cat /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/tags/list  | jq .
{
  "name": "rhosp-rhel8/openstack-aodh-listener",
  "tags": [
    "16.1.4",
    "16.1"
  ]
}
[stack@director ~]$ ll /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/*.type-map
-rw-r--r--. 1 root root 169 Nov 25 11:01 /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4.type-map
-rw-r--r--. 1 root root 167 Nov 25 11:40 /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.type-map
[stack@director ~]$ 

5. On one of the controllers, check for the current version of the same image:

[root@controller1-161 ~]# podman images |grep aodh
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1.4       d6480804c4d2   8 months ago   743 MB
[root@controller1-161 ~]# 


6. Pull the image using the tag 16.1:

[root@controller1-161 ~]# podman pull director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator:16.1
Trying to pull director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator:16.1...
Getting image source signatures
Copying blob ea462113cc1f skipped: already exists
Copying blob fe9c8cc943e3 skipped: already exists
Copying blob 4422ea938fce skipped: already exists
Copying blob 00f18994d663 skipped: already exists
Copying blob d8ec3e673e1e skipped: already exists
Copying blob 505209649de3 skipped: already exists
Copying config d6480804c4 done
Writing manifest to image destination
Storing signatures
d6480804c4d2cb600479b46f92f728db804410d68c264566dd5d3859a21bd750
[root@controller1-161 ~]# 


Actual results:

As a result, the controller has pulled the same image than already had, and added the tag 16.1 to it:

[root@controller1-161 ~]# podman images |grep aodh
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1         d6480804c4d2   8 months ago   743 MB
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1.4       d6480804c4d2   8 months ago   743 MB
[root@controller1-161 ~]# 


When using curl to check the images manifest, they both show the same:

[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1 | jq .config.digest
"sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273"
[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4 | jq .config.digest
"sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273"
[root@controller1-161 ~]# 


However, the Apache type-map file correctly shows different digest:

[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.type-map | jq .config.digest
"sha256:ea548ddc17674629c97f7472cb5fe62f5495208bcf2f27d5918bd9a8e7c9833f"
[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4.type-map | jq .config.digest
"sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273"
[root@controller1-161 ~]# 

Therefore, the type-map files have the correct content, but Apache is always using the older one.


Expected results:

Apache type-map should correctly map the GET request of */16.1, to the type-map file */16.1.type-map, instead of matching 16.1.4.type-map, so that podman can pull the right image.


Additional info:


[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/keeping_red_hat_openstack_platform_updated/index#updating-your-container-image-preparation-file_keeping-updated