(In reply to Marius Cornea from comment #0)
> Actual results:
> Deployment fails due to missing images.

Missing tags to be exact.
 
> Expected results:
> Image tags detection is done correctly and deployment can find all required
> images.

Agree, but let's clarify why this is happening. We can't assume external projects which integrate with TripleO will follow the same tagging conventions. Though the registry for the ceph image can be queried for its version and it's release (e.g. see 4 and 122 in [1]), it is not the convention of the maintainer of that registry to create tags with a string made from '{version}-{release}'. 

Since ceph already has exceptions with the following defaults:

      ceph_image: daemon
      ceph_namespace: docker.io/ceph
      ceph_tag: v4.0.0-stable-4.0-nautilus-centos-7-x86_64

It probably makes sense to document upstream [2] and downstream with a docbug that "tag_from_label: '{version}-{release}'" does not work for Ceph containers and to modify tripleo-common to not use that tag when using the ceph container.


[1] 
[INSPECTION DATA]: {u'io.k8s.description': u'The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.', u'maintainer': u'Sebastien Han <seb>', u'com.redhat.license_terms': u'https://www.redhat.com/licenses/eulas', u'vcs-ref': u'd5ff1490fad8f1b57e451d384d3b331e94cf6fe4', u'authoritative-source-url': u'registry.access.redhat.com', u'io.k8s.display-name': u'Red Hat Ceph Storage 4 on RHEL 8', u'version': u'4', u'GIT_CLEAN': u'True', u'com.redhat.component': u'rhceph-container', u'distribution-scope': u'public', u'vendor': u'Red Hat, Inc.', u'description': u'Red Hat Ceph Storage 4', u'vcs-type': u'git', u'CEPH_POINT_RELEASE': u'', u'GIT_BRANCH': u'HEAD', u'GIT_COMMIT': u'9b488353828a3dfe71ca42df6595abfacf13bad2', u'RELEASE': u'HEAD', u'build-date': u'2019-04-25T16:26:03.051400', u'name': u'rhceph', u'url': u'https://access.redhat.com/con
tainers/#/registry.access.redhat.com/ubi8/images/8.0-122', u'summary': u'Provides the latest Red Hat Ceph Storage 4 on RHEL 8 in a fully featured and supported base image.', u'architecture': u'x86_64', u'release': u'122', u'io.openshift.expose-services': u'', u'io.openshift.tags': u'rhceph ceph', u'GIT_REPO': u'https://github.com/ceph/ceph-container', u'com.redhat.build-host': u'cpt-0013.osbs.prod.upshift.rdu2.redhat.com'}

Exception: Image docker-registry.upshift.redhat.com/ceph/rhceph-4.0-rhel8 has no tag 4-122.
Available tags: latest

[2] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/container_image_prepare.html#default-registry

If the registry docker-registry.upshift.redhat.com is just a pre-release registry and this image will eventually be served by registry.access.redhat.com with the standard tagging policy, then I would suggest changing your ContainerImagePrepare value into multiple entries so the ceph tagging policy can be properly represented. Something like this should work:

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: true
    set:
      name_prefix: default_organization-osp15_containers-
      name_suffix: ''
      namespace: rhos-compute-node-08.lab.eng.rdu2.redhat.com:5000
      neutron_driver: ovn
      openshift_asb_namespace: registry.access.redhat.com/openshift3
      openshift_asb_tag: v3.11
      openshift_cluster_monitoring_image: ose-cluster-monitoring-operator
      openshift_cluster_monitoring_namespace: registry.access.redhat.com/openshift3
      openshift_cluster_monitoring_tag: v3.11
      openshift_cockpit_image: registry-console
      openshift_cockpit_namespace: registry.access.redhat.com/openshift3
      openshift_cockpit_tag: v3.11
      openshift_configmap_reload_image: ose-configmap-reloader
      openshift_configmap_reload_namespace: registry.access.redhat.com/openshift3
      openshift_configmap_reload_tag: v3.11
      openshift_etcd_image: etcd
      openshift_etcd_namespace: registry.access.redhat.com/rhel7
      openshift_etcd_tag: latest
      openshift_gluster_block_image: rhgs-gluster-block-prov-rhel7
      openshift_gluster_image: rhgs-server-rhel7
      openshift_gluster_namespace: registry.access.redhat.com/rhgs3
      openshift_gluster_tag: v3.11
      openshift_grafana_namespace: registry.access.redhat.com/openshift3
      openshift_grafana_tag: v3.11
      openshift_heketi_image: rhgs-volmanager-rhel7
      openshift_heketi_namespace: registry.access.redhat.com/rhgs3
      openshift_heketi_tag: v3.11
      openshift_kube_rbac_proxy_image: ose-kube-rbac-proxy
      openshift_kube_rbac_proxy_namespace: registry.access.redhat.com/openshift3
      openshift_kube_rbac_proxy_tag: v3.11
      openshift_kube_state_metrics_image: ose-kube-state-metrics
      openshift_kube_state_metrics_namespace: registry.access.redhat.com/openshift3
      openshift_kube_state_metrics_tag: v3.11
      openshift_namespace: registry.access.redhat.com/openshift3
      openshift_oauth_proxy_tag: v3.11
      openshift_prefix: ose
      openshift_prometheus_alertmanager_tag: v3.11
      openshift_prometheus_config_reload_image: ose-prometheus-config-reloader
      openshift_prometheus_config_reload_namespace: registry.access.redhat.com/openshift3
      openshift_prometheus_config_reload_tag: v3.11
      openshift_prometheus_node_exporter_tag: v3.11
      openshift_prometheus_operator_image: ose-prometheus-operator
      openshift_prometheus_operator_namespace: registry.access.redhat.com/openshift3
      openshift_prometheus_operator_tag: v3.11
      openshift_prometheus_tag: v3.11
      openshift_tag: v3.11
      tag: 15.0
    tag_from_label: '{version}-{release}'
    excludes: [ceph]
  - push_destination: true
    set:
      ceph_image: rhceph-4.0-rhel8
      ceph_namespace: docker-registry.upshift.redhat.com/ceph
      ceph_tag: latest
    includes: [ceph]

This sort of approach is documented upstream at [1]

[1] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/container_image_prepare.html#layering-image-preparation-entries

Marius,

Based on Steve's comment #2 do you want to try the proposed syntax and if it works for you to solve this problem, then we can turn this into a docbug so that our docs have a similar example?

  John

(In reply to John Fulton from comment #3)
> Marius,
> 
> Based on Steve's comment #2 do you want to try the proposed syntax and if it
> works for you to solve this problem, then we can turn this into a docbug so
> that our docs have a similar example?
> 
>   John

I think we need to have the generated containers-prepare-parameter.yaml work out of the box with the official Red hat Registry(whichever that will be at GA time). We shouldn't require the user to do any extra manual steps to edit containers-prepare-parameter.yaml. Meanwhile we can consider manually editing containers-prepare-parameter.yaml as a workaround.

(In reply to Marius Cornea from comment #5)
> (In reply to John Fulton from comment #3)
> > Marius,
> > 
> > Based on Steve's comment #2 do you want to try the proposed syntax and if it
> > works for you to solve this problem, then we can turn this into a docbug so
> > that our docs have a similar example?
> > 
> >   John
> 
> I think we need to have the generated containers-prepare-parameter.yaml work
> out of the box with the official Red hat Registry(whichever that will be at
> GA time). We shouldn't require the user to do any extra manual steps to edit
> containers-prepare-parameter.yaml. Meanwhile we can consider manually
> editing containers-prepare-parameter.yaml as a workaround.

Only two ways to do that:

1. Ceph adopts the convention of tagging with '{version}-{release}' (needinfo Ken, what do you think of this?)
2. We make the defaults in the generated containers-prepare-parameter.yaml have the exclude list

  John

I think I see the problem.

OSBS inserts (and increments) a "release" label automatically whenever it builds a container that lacks that label.

We don't build upstream Ceph containers through OSBS at all, so we cannot expect to ever have a "release" label there. You'll need to work around this upstream for the forseeable future.

Downstream, Thomas and I usually build containers through OSBS. However, we took some shortcuts with RH Ceph Storage 4, and I'm not building containers in OSBS there yet.

This means we're inheriting the "122" release label from our UBI base image, whoops.

We are close to paying off those shortcuts' technical debt, so we can build Ceph in OSBS. Here is my first attempt from yesterday: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=21707920 .This failed and we need to fix https://github.com/ceph/ceph-container/issues/1370 to advance there.

Secondly, once we're building in OSBS all the time, I'll need to update our Jenkins job on ceph-jenkins to push this second tag into docker-registry.upshift.redhat.com.

(In reply to John Fulton from comment #6)
> (In reply to Marius Cornea from comment #5)
> > (In reply to John Fulton from comment #3)
> > > Marius,
> > > 
> > > Based on Steve's comment #2 do you want to try the proposed syntax and if it
> > > works for you to solve this problem, then we can turn this into a docbug so
> > > that our docs have a similar example?
> > > 
> > >   John
> > 
> > I think we need to have the generated containers-prepare-parameter.yaml work
> > out of the box with the official Red hat Registry(whichever that will be at
> > GA time). We shouldn't require the user to do any extra manual steps to edit
> > containers-prepare-parameter.yaml. Meanwhile we can consider manually
> > editing containers-prepare-parameter.yaml as a workaround.
> 
> Only two ways to do that:
> 
> 1. Ceph adopts the convention of tagging with '{version}-{release}'
> (needinfo Ken, what do you think of this?)
> 2. We make the defaults in the generated containers-prepare-parameter.yaml
> have the exclude list

As per bz 1525231 and Ken's reply in #7, "no" to #1 above.

That leaves #2, do we need to change defaults? 

Well, only if our defaults ship with '{version}-{release}'? If they don't, then the defaults won't break anything when Ceph is involved. Let's find out.

I encountered this issue in OSP16 beta release, so you should fix it in OSP16 too.