Bug 1741247 - [Upgrades/updates] External update/upgrade run --tags container_image_prepare does not upload the container images in the Undercloud
Summary: [Upgrades/updates] External update/upgrade run --tags container_image_prepare...
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 15.0 (Stein)
Assignee: Jose Luis Franco
QA Contact: Raviv Bar-Tal
: 1746440 (view as bug list)
Depends On: 1746815
Blocks: 1727807 1727808
TreeView+ depends on / blocked
Reported: 2019-08-14 15:10 UTC by Jose Luis Franco
Modified: 2019-09-27 10:44 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-validations-10.5.1-0.20190830060432.585c119.el8ost python-tripleoclient-11.5.1-0.20190829110437.9b9b5aa.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-09-21 11:24:25 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1832932 0 None None None 2019-08-14 15:14:29 UTC
OpenStack gerrit 677777 0 'None' MERGED Add undercloud-key-file and ssh_network options. 2020-08-21 09:21:44 UTC
OpenStack gerrit 679054 0 'None' MERGED Pass undercloud's private key to inventory used in package_update. 2020-08-21 09:21:44 UTC
OpenStack gerrit 679061 0 'None' MERGED Pass undercloud_connection in TripleoInventory instantiation. 2020-08-21 09:21:44 UTC
OpenStack gerrit 679243 0 'None' MERGED Pass undercloud_connection in TripleoInventory instantiation. 2020-08-21 09:21:44 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:25:00 UTC

Description Jose Luis Franco 2019-08-14 15:10:28 UTC
Description of problem:

Since OSP14 the way to provide the container images to be used during deployment/update/upgrade passed to be specified via the ContainerImagePrepare environment parameter.

When upgrading, it was this command the one that allowed us to point to the new version's container images:

openstack external-upgrade run --tags container_image_prepare

In a similar way,the container images were updating via:

openstack external-update run --tags container_image_prepare

In OSP15 these two commands seemed to be running, however when executing the upgrade/update workflow the container images were not present in the local registry (now a httpd vhost running in the Undercloud named image-serve). As a consequence, as the overcloud images could not be retrieved the update/upgrade process failed.

The tripleo-container-image-prepare command is being executed in the mistral_executor container locally, instead of on the Undercloud's host. Therefore, if we display the content of the /var/lib/image-serve/v2/rhosp15 folder in the mistral_exector, we will see the container images we want to update/upgrade to.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Deploy OSP15
2. Run the undercloud upgrade
3. Run the overcloud update prepare passing the new containers_prepare_parameters.yaml (pointing to the tag we want to update to)
4. Run the "openstack external-update run --tags container_image_prepare" command to upload the container images.
5. Print out the content of the Undercloud's directory /var/lib/image-serve/v2/rhosp15 and see that all the overcloud service container images are missing the new tag which should have been pulled.
6. Display mistral_exector image-serve folder via "sudo podman exec -it mistral_executor ls /var/lib/image-serve/v2/rhosp15"
7. All the images missing in the Undercloud can be found there (or the newest tag in the case of an update).

Actual results:

The container images are being up

Expected results:

Additional info:

Comment 1 Jose Luis Franco 2019-08-14 15:16:18 UTC
A way to workaround this issue is to execute the old openstack command to upload the container images:

sudo openstack tripleo container image prepare \
  -e ~/containers-prepare-parameter.yaml

Comment 2 Sofer Athlan-Guyot 2019-08-28 13:32:42 UTC
*** Bug 1746440 has been marked as a duplicate of this bug. ***

Comment 6 Sofer Athlan-Guyot 2019-08-30 07:34:22 UTC
Everything is merged in stein.

Comment 15 Jose Luis Franco 2019-09-09 07:43:30 UTC
Hello Greg,

In theory this change was meant to fix something which was already part of the upgrade/update process, so no extra doc would be needed besides the one we already had from OSP14: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html/keeping_red_hat_openstack_platform_updated/assembly-updating_the_overcloud#running-the-container-image-preparation

Comment 17 errata-xmlrpc 2019-09-21 11:24:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.