Bug 1522631

Summary: docker images have multiple tags so pulling them as per documentation doesn't work
Product: Red Hat OpenStack Reporter: Anil Dhingra <adhingra>
Component: openstack-tripleo-commonAssignee: Steve Baker <sbaker>
Status: CLOSED ERRATA QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact: Andrew Burden <aburden>
Priority: high    
Version: 12.0 (Pike)CC: adhingra, agurenko, asimonel, emacchi, jboggs, jschluet, m.andre, mburns, mburrows, rhallise, sasha, sbaker, slinaber, tvignaud
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-common-8.5.1-0.20180326153322.91f52e9.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:40:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Anil Dhingra 2017-12-06 06:48:58 UTC
Description of problem:

docker images have multiple tags so pulling them as per documentation doesn't work

after following below sequence container_images.yaml have images with label 12.0-20171129.1 but at some images still have 12.0-20171103.1 so upload didnt work

(undercloud) [stack@undercloud ~]$ openstack overcloud container image tag discover --image registry.access.redhat.com/rhosp12-beta/openstack-base:latest --tag-from-label version-release
12.0-20171129.1

(undercloud) [stack@undercloud ~]$ openstack overcloud container image prepare \
>   --namespace=registry.access.redhat.com/rhosp12-beta \
>   --prefix=openstack- \
>   --tag=12.0-20171129.1 \
>   --set ceph_namespace=registry.access.redhat.com/rhosp12-beta \
>   --set ceph_image=rhceph-2-rhel7 \
>   --set ceph_tag=latest > /home/stack/container_images.yaml


below command fails as label mismatch 

undercloud) [stack@undercloud ~]$ sudo openstack overcloud container image upload --verbose --config-file /home/stack/container_images.yaml

---
Could not pull image registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
END return value: 1

---
(undercloud) [stack@undercloud ~]$ grep -i gnocc container_images.yaml                                                                                                                                                                      
- imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api:12.0-20171129.1
- imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-metricd:12.0-20171129.1
- imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-statsd:12.0-20171129.1

but on registry.access.redhat.com/rhosp12-beta label is 12.0-20171103.1 for multiple services

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

all images should have same tag to 
Expected results:


Additional info:

Comment 4 Alexander Chuzhoy 2017-12-06 15:02:21 UTC
Reproduced.


(undercloud) [stack@undercloud-0 ~]$  sudo openstack overcloud container image upload --verbose --config-file /home/stack/container_images.yaml
START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/container_images.yaml']
command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)
Using config files: [u'/home/stack/container_images.yaml']
imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-api:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-api:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-evaluator:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-evaluator:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-listener:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-listener:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-notifier:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-notifier:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-central:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-central:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-compute:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-compute:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-notification:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-notification:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-cron:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-cron:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-glance-api:12.0-20171129.1
Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-glance-api:12.0-20171129.1
imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api:12.0-20171129.1
docker pull failed: unknown: Not Found
retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
docker pull failed: unknown: Not Found
retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
docker pull failed: unknown: Not Found
retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
docker pull failed: unknown: Not Found
retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
docker pull failed: unknown: Not Found
retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
Could not pull image registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api
END return value: 1

Seems like the tag need to be changed on some images - at least the following (currently set to 12.0-20171103.1):

openstack-haproxy
openstack-gnocchi-api
openstack-gnocchi-metricd 
openstack-heat-api-cfn
openstack-heat-api
openstack-heat-engine
openstack-horizon

Comment 5 Steve Baker 2017-12-06 20:20:51 UTC
For OSP-12 whenever images are rebuilt and pushed to the container catalog, it will need to be a *full* rebuild with all images being pushed with a consistent tag (12.0-20171129.1).

Also I would recommend that the "latest" tag only be applied to the images after all of the 12.0-20171129.1 images are uploaded to the catalog, to avoid races where people deploy with a particular version but not all images for that version exist in the catalog yet.

I'm setting a NEEDINFO on Jon Schlueter to confirm if the above 2 build pipeline requirements are in place.

In OSP-13 the prepare tooling will be able to inspect every latest image, so that a mixture of versioned tags can be discovered.

Comment 11 Steve Baker 2018-03-26 22:55:18 UTC
The fix for this is ready for testing in OSP13 by following the instructions in this doc change:

https://gitlab.cee.redhat.com/sbaker/docs-Red_Hat_Enterprise_Linux_OpenStack_Platform/commit/9ae8d5481fde7f4e21d9891786be3dc61568a60b

Basically, the command "openstack overcloud container image tag discover" should never be called, and any call to "openstack overcloud container image prepare" needs the following argument:

  --tag-from-label {version}-{release}

This will result in discovery happening on every image (including ceph), allowing individual images to have different versioned tags.

Comment 20 Gurenko Alex 2018-06-06 14:59:57 UTC
--tag-from-label {version}-{release} works as expected, each container recieves its own tag.

Comment 22 errata-xmlrpc 2018-06-27 13:40:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086