Description of problem: docker images have multiple tags so pulling them as per documentation doesn't work after following below sequence container_images.yaml have images with label 12.0-20171129.1 but at some images still have 12.0-20171103.1 so upload didnt work (undercloud) [stack@undercloud ~]$ openstack overcloud container image tag discover --image registry.access.redhat.com/rhosp12-beta/openstack-base:latest --tag-from-label version-release 12.0-20171129.1 (undercloud) [stack@undercloud ~]$ openstack overcloud container image prepare \ > --namespace=registry.access.redhat.com/rhosp12-beta \ > --prefix=openstack- \ > --tag=12.0-20171129.1 \ > --set ceph_namespace=registry.access.redhat.com/rhosp12-beta \ > --set ceph_image=rhceph-2-rhel7 \ > --set ceph_tag=latest > /home/stack/container_images.yaml below command fails as label mismatch undercloud) [stack@undercloud ~]$ sudo openstack overcloud container image upload --verbose --config-file /home/stack/container_images.yaml --- Could not pull image registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api END return value: 1 --- (undercloud) [stack@undercloud ~]$ grep -i gnocc container_images.yaml - imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api:12.0-20171129.1 - imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-metricd:12.0-20171129.1 - imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-statsd:12.0-20171129.1 but on registry.access.redhat.com/rhosp12-beta label is 12.0-20171103.1 for multiple services Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: all images should have same tag to Expected results: Additional info:
Reproduced. (undercloud) [stack@undercloud-0 ~]$ sudo openstack overcloud container image upload --verbose --config-file /home/stack/container_images.yaml START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/container_images.yaml'] command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False) Using config files: [u'/home/stack/container_images.yaml'] imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-api:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-api:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-evaluator:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-evaluator:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-listener:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-listener:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-aodh-notifier:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-aodh-notifier:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-central:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-central:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-compute:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-compute:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-notification:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-ceilometer-notification:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-cron:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-cron:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-glance-api:12.0-20171129.1 Completed upload for docker image registry.access.redhat.com/rhosp12-beta/openstack-glance-api:12.0-20171129.1 imagename: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api:12.0-20171129.1 docker pull failed: unknown: Not Found retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api docker pull failed: unknown: Not Found retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api docker pull failed: unknown: Not Found retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api docker pull failed: unknown: Not Found retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api docker pull failed: unknown: Not Found retrying pulling image: registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api Could not pull image registry.access.redhat.com/rhosp12-beta/openstack-gnocchi-api END return value: 1 Seems like the tag need to be changed on some images - at least the following (currently set to 12.0-20171103.1): openstack-haproxy openstack-gnocchi-api openstack-gnocchi-metricd openstack-heat-api-cfn openstack-heat-api openstack-heat-engine openstack-horizon
For OSP-12 whenever images are rebuilt and pushed to the container catalog, it will need to be a *full* rebuild with all images being pushed with a consistent tag (12.0-20171129.1). Also I would recommend that the "latest" tag only be applied to the images after all of the 12.0-20171129.1 images are uploaded to the catalog, to avoid races where people deploy with a particular version but not all images for that version exist in the catalog yet. I'm setting a NEEDINFO on Jon Schlueter to confirm if the above 2 build pipeline requirements are in place. In OSP-13 the prepare tooling will be able to inspect every latest image, so that a mixture of versioned tags can be discovered.
The fix for this is ready for testing in OSP13 by following the instructions in this doc change: https://gitlab.cee.redhat.com/sbaker/docs-Red_Hat_Enterprise_Linux_OpenStack_Platform/commit/9ae8d5481fde7f4e21d9891786be3dc61568a60b Basically, the command "openstack overcloud container image tag discover" should never be called, and any call to "openstack overcloud container image prepare" needs the following argument: --tag-from-label {version}-{release} This will result in discovery happening on every image (including ceph), allowing individual images to have different versioned tags.
--tag-from-label {version}-{release} works as expected, each container recieves its own tag.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086