Bug 1756443

Created attachment 1620211 [details]
Logs

Description of problem:

There was a tag mismatch happen between what upgrade run yaml scripts trying to pull and what actually available in UC's registry.

Fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "pull", "192.168.24.1:8787/rhosp15/openstack-cinder-volume:15.0-70"]

VS

curl -X GET http://192.168.24.1:8787/v2/rhosp15/openstack-cinder-volume/{"name": "rhosp15/openstack-cinder-volume", "tags": ["15.0-71"]


Version-Release number of selected component(if applicable):
(14->15 upgrade)

How reproducible:
It occurs at irregular interval, faced by two-three engineers as of my knowledge. But worked fine after re running the overcloud upgrade prepare step again.

Steps to Reproduce:

Use the below commands:

cat <<EOF > ~/overcloud-params.yaml
parameter_defaults:
  UpgradeLeappDevelSkip: LEAPP_DEVEL_SKIP_RHSM=1 LEAPP_SKIP_CHECK_OS_RELEASE=1
  UpgradeLeappDevelSkipRhsm: true
  SELinuxMode: permissive
  ContainerHealthcheckDisabled: true
EOF

cp overcloud_deploy.sh overcloud_upgrade_prepare.sh
sed -i overcloud_upgrade_prepare.sh -e 's/openstack overcloud deploy/OVERCLOUD_SKIP_UPGRADE_SUPPORT_CHECK=1 openstack overcloud upgrade prepare/g'
sed -i overcloud_upgrade_prepare.sh -e '/\-e ~\/containers-prepare-parameter.yaml/i -e \/usr\/share\/openstack-tripleo-heat-templates\/environments\/services\/neutron-ovs.yaml \\'
sed -i overcloud_upgrade_prepare.sh -e '/\-e ~\/containers-prepare-parameter.yaml/a -e ~\/overcloud-params\.yaml \\'

bash -x overcloud_upgrade_prepare.sh 2>&1 | tee oc-upgrade-prepare.log

# Upload images to local registry on undercloud
openstack overcloud external-upgrade run --tags container_image_prepare 2>&1 | tee oc-container-image-prepare.log


# check the stack environment Tag Both will shown  different tag(15.0-70 vs 15.0-71]:

Fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "pull", "192.168.24.1:8787/rhosp15/openstack-cinder-volume:15.0-70"]

VS

curl -X GET http://192.168.24.1:8787/v2/rhosp15/openstack-cinder-volume/{"name": "rhosp15/openstack-cinder-volume", "tags": ["15.0-71"]

 
Actual results:
Pull latest cinder_volume images - failed

Expected results:
Pull latest cinder_volume images - success

Additional info:
On 2019-09-26 22:36, Jose Luis Franco Arza wrote:
> Ok, I re-ran the "overcloud upgrade prepare" command and it now takes the
> right tag in the heat stack:
> 
> (undercloud) [stack@undercloud-0 ~]$ openstack stack environment show
> overcloud | grep "rhosp15/openstack-cinder-volume"
>    ContainerCinderVolumeImage:
> 192.168.24.1:8787/rhosp15/openstack-cinder-volume:15.0-71
>    DockerCinderVolumeImage:
> 192.168.24.1:8787/rhosp15/openstack-cinder-volume:15.0-71
> 
> (undercloud) [stack@undercloud-0 ~]$ curl -X GET
> http://192.168.24.1:8787/v2/rhosp15/openstack-cinder-volume/tags/list
> {"name": "rhosp15/openstack-cinder-volume", "tags": ["15.0-71"]}
> 
> Try to run the overcloud upgrade run again, please.

I've hit a similar thing earlier in my testing, there may be a real bug 
here.

Not 100% sure but i think this may be happening: the `openstack 
overcloud upgrade prepare` command looks at latest images and writes 
*ContainerImage Heat parameters, then `openstack external-upgrade run 
--tags container_image_prepare` *also* looks at latest images and 
fetches them. So if the definition of what's latest image changes 
between running those two commands (e.g. new containers happen to be 
released), then there can be a mismatch between what's in params and 
what's in UC registry.

Maybe the same thing might happen on `overcloud deploy` too, the chance 
is much lower because both pieces are executed within a same CLI 
command, but i suspect the race condition might still be there.
The internals of the image parameters/uploads are more of a DFG:DF 
expertise so we will probably ask them for some insight. Ideally the 
image uploader would look at the generated parameters and fetch those 
images instead of fetching latest, but i'm not sure how easy that fix 
would be.

The workaround in our case is to always run both of the aforementioned 
commands in immediate succession.