Bug 2028962 - Undercloud fails to provide correct images to overcloud when updating tags from 16.1.x to 16.1
Summary: Undercloud fails to provide correct images to overcloud when updating tags fr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: OSP Team
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On: 2026654
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-03 20:21 UTC by Alex Schultz
Modified: 2022-03-23 22:12 UTC (History)
6 users (show)

Fixed In Version: tripleo-ansible-0.8.1-2.20211216164949.1fdb766.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2026654
Environment:
Last Closed: 2022-03-23 22:12:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 820594 0 None NEW Disable image-serve MultiViews 2021-12-06 21:35:37 UTC
Red Hat Issue Tracker OSP-11179 0 None None None 2021-12-03 20:25:50 UTC
Red Hat Product Errata RHBA-2022:1001 0 None None None 2022-03-23 22:12:53 UTC

Description Alex Schultz 2021-12-03 20:21:31 UTC
This would also affect 16.2 and 16.2.1

+++ This bug was initially created as a clone of Bug #2026654 +++

Description of problem:
When updating the overcloud, the undercloud registry fails to provide the correct images, if the following conditions are met:

1. Used containers-prepare-parameter.yaml with 16.1.x tag (with x < 6, as of this writing) to perform the initial deployment of RHOSP
2. Used containers-prepare-parameter.yaml with 16.1 tag when preparing the udpate

This causes the MultiView handler on apache to always return the older image, even if the 16.1 tag is specified:

[root@controller1-161 ~]# podman images |grep aodh
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1         d6480804c4d2   8 months ago   743 MB
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1.4       d6480804c4d2   8 months ago   743 MB


How reproducible:
Always reproducible by pulling images from CDN to prepare overcloud upgrade, if tag 16.1.4 is used first, and then 16.1.

Steps to Reproduce:
1. Deploy undercloud and overcloud using tag 16.1.4 on the file containers-prepare-parameter.yaml

$ egrep -B1 tag_from_label containers-prepare-parameter.yaml
      tag: '16.1.4'
    tag_from_label: '{version}-{release}'

2. Change tag from 16.1.4 to 16.1, in preparation for update, and as documented on [1]

$ sed -i 's/16.1.4/16.1/' containers-prepare-parameter.yaml
$ egrep -B1 tag_from_label containers-prepare-parameter.yaml
      tag: '16.1'
    tag_from_label: '{version}-{release}'

3. Update undercloud image registry

$ sudo openstack tripleo container image prepare -e containers-prepare-parameter.yaml 

4. Test for a given image, that the version with tag 16.1 has been populated on the undercloud registry (along with the older one):

[stack@director ~]$ cat /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/tags/list  | jq .
{
  "name": "rhosp-rhel8/openstack-aodh-listener",
  "tags": [
    "16.1.4",
    "16.1"
  ]
}
[stack@director ~]$ ll /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/*.type-map
-rw-r--r--. 1 root root 169 Nov 25 11:01 /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4.type-map
-rw-r--r--. 1 root root 167 Nov 25 11:40 /var/lib/image-serve/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.type-map
[stack@director ~]$ 

5. On one of the controllers, check for the current version of the same image:

[root@controller1-161 ~]# podman images |grep aodh
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1.4       d6480804c4d2   8 months ago   743 MB
[root@controller1-161 ~]# 


6. Pull the image using the tag 16.1:

[root@controller1-161 ~]# podman pull director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator:16.1
Trying to pull director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator:16.1...
Getting image source signatures
Copying blob ea462113cc1f skipped: already exists
Copying blob fe9c8cc943e3 skipped: already exists
Copying blob 4422ea938fce skipped: already exists
Copying blob 00f18994d663 skipped: already exists
Copying blob d8ec3e673e1e skipped: already exists
Copying blob 505209649de3 skipped: already exists
Copying config d6480804c4 done
Writing manifest to image destination
Storing signatures
d6480804c4d2cb600479b46f92f728db804410d68c264566dd5d3859a21bd750
[root@controller1-161 ~]# 


Actual results:

As a result, the controller has pulled the same image than already had, and added the tag 16.1 to it:

[root@controller1-161 ~]# podman images |grep aodh
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1         d6480804c4d2   8 months ago   743 MB
director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-aodh-evaluator       16.1.4       d6480804c4d2   8 months ago   743 MB
[root@controller1-161 ~]# 


When using curl to check the images manifest, they both show the same:

[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1 | jq .config.digest
"sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273"
[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4 | jq .config.digest
"sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273"
[root@controller1-161 ~]# 


However, the Apache type-map file correctly shows different digest:

[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.type-map | jq .config.digest
"sha256:ea548ddc17674629c97f7472cb5fe62f5495208bcf2f27d5918bd9a8e7c9833f"
[root@controller1-161 ~]# curl -s http://director.ctlplane.localdomain:8787/v2/rhosp-rhel8/openstack-aodh-listener/manifests/16.1.4.type-map | jq .config.digest
"sha256:6d6130ebb9f4c6b0ad2b1d9a29c9d2fa84df24afc9b4744e5232f75346cd4273"
[root@controller1-161 ~]# 

Therefore, the type-map files have the correct content, but Apache is always using the older one.


Expected results:

Apache type-map should correctly map the GET request of */16.1, to the type-map file */16.1.type-map, instead of matching 16.1.4.type-map, so that podman can pull the right image.


Additional info:


[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/keeping_red_hat_openstack_platform_updated/index#updating-your-container-image-preparation-file_keeping-updated

Comment 1 Eric Nothen 2022-01-12 08:40:34 UTC
Is it to early to ask on which z stream is this fix going to be included?

Comment 4 David Rosenfeld 2022-02-01 20:32:08 UTC
Procedure used was:

deploy undercloud with puddle that has fix
openstack tripleo container image prepare with a tag of 16.2.1
podman pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator
openstack tripleo container image prepare with a tag of 16.2
podman pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator

Saw that 16.2.1 and 16.2 ids were different:

[stack@undercloud-0 ~]$ sudo podman images | grep aodh
undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator             16.2                    2548ab62702d  3 hours ago   700 MB
undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-aodh-evaluator             16.2.1                  ce85155e7718  2 months ago  701 MB
[stack@undercloud-0 ~]$

Comment 12 errata-xmlrpc 2022-03-23 22:12:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1001


Note You need to log in before you can comment on or make changes to this bug.