Bug 1726171

cat containers-prepare-parameter.yaml 
# Generated with the following on 2019-07-01T11:09:16.313279
#
#   openstack tripleo container image prepare default --output-env-file /home/stack/containers-prepare-parameter.yaml --local-push-destination
#

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: true
    set:
      ceph_image: rhceph-4-rhel8
      ceph_namespace: registry.redhat.io/rhceph-beta
      ceph_tag: latest
      name_prefix: openstack-
      name_suffix: ''
      namespace: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15
      neutron_driver: ovn
      tag: 20190625.1
/var/log/tripleo-container-image-prepare.log in attachment

Created attachment 1586605 [details]
/var/log/tripleo-container-image-prepare.log

/var/log/tripleo-container-image-prepare.log

ansible-playbook   -i inventory.yaml   --private-key ~/.ssh/id_rsa   --become   ~/config-download/deploy_steps_playbook.yaml --skip-tags container_image_prepare didn't help

Artem,

This isn't really a bug. You just need to pass the correct parameters. I'll explain.

The error you got is here:

imagename: registry.redhat.io/rhceph-beta/rhceph-4-rhel8:latest
Image prepare failed: 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Arhceph-beta%2Frhceph-4-rhel8%3Apull

This is because you need a Red Hat account to access our container images. To use your Red Hat account to access the Red Hat images please see this docbug

 https://bugzilla.redhat.com/show_bug.cgi?id=1723969

*** This bug has been marked as a duplicate of bug 1723969 ***

(In reply to John Fulton from comment #4)
> Artem,
> 
> This isn't really a bug. You just need to pass the correct parameters. I'll
> explain.
> 
> The error you got is here:
> 
> imagename: registry.redhat.io/rhceph-beta/rhceph-4-rhel8:latest
> Image prepare failed: 401 Client Error: Unauthorized for url:
> https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/
> auth?service=docker-registry&scope=repository%3Arhceph-beta%2Frhceph-4-
> rhel8%3Apull
> 
> This is because you need a Red Hat account to access our container images.
> To use your Red Hat account to access the Red Hat images please see this
> docbug
> 
>  https://bugzilla.redhat.com/show_bug.cgi?id=1723969
> 

Can we provide a pre-deployment validation that ensures ceph credentials were provided and do not start the deployment when credentials are not present? If the ceph registry now requires authentication then imho we shouldn't start deployment if credentials were not provided.

(In reply to Marius Cornea from comment #6)
> (In reply to John Fulton from comment #4)
> > Artem,
> > 
> > This isn't really a bug. You just need to pass the correct parameters. I'll
> > explain.
> > 
> > The error you got is here:
> > 
> > imagename: registry.redhat.io/rhceph-beta/rhceph-4-rhel8:latest
> > Image prepare failed: 401 Client Error: Unauthorized for url:
> > https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/
> > auth?service=docker-registry&scope=repository%3Arhceph-beta%2Frhceph-4-
> > rhel8%3Apull
> > 
> > This is because you need a Red Hat account to access our container images.
> > To use your Red Hat account to access the Red Hat images please see this
> > docbug
> > 
> >  https://bugzilla.redhat.com/show_bug.cgi?id=1723969
> > 
> 
> Can we provide a pre-deployment validation that ensures ceph credentials
> were provided and do not start the deployment 

I expect most customers will store the container registry on their undercloud by running a command like the following before they run 'openstack overcloud deploy':

 sudo openstack tripleo container image prepare -e containers.yaml --output-env-file containers-env-file.yaml

The above downloads the containers from the external Red Hat registry (using credentials in containers.yaml) and sets them up to be served from the undercloud. Then the overcloud nodes get the containers from the undercloud, provided that the overcloud is deployed with '-e containers-env-file.yaml'.

When the customer follows this pattern without providing their credentials they'll fail the authentication before the overcloud deploy and get the feedback early. This the pattern that's recommended recommended in our docs. See section 2.1:

 https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html/transitioning_to_containerized_services/obtaining-container-images

The OSP15 version of the above will be updated with how to use ContainerImageRegistryCredentials as per docbug 1723969.

> when credentials are not
> present? If the ceph registry now requires authentication then imho we
> shouldn't start deployment if credentials were not provided.

They're not exactly ceph credentials, they're Red Hat registry credentials and I expect in OSP16 that the OpenStack containers will also be hosted on registry.redhat.io which will continue to require authentication.

This issue isn't related to Ceph integration. It's related to director's overall process for obtaining container images and the docs will be updated to cover accessing our containers from our new registry. If you still think we need additional validations then perhaps this should be moved to an RFE so PM can assess it and it can be scheduled for a future feature.

(In reply to John Fulton from comment #7)
> (In reply to Marius Cornea from comment #6)
> > (In reply to John Fulton from comment #4)
> > > Artem,
> > > 
> > > This isn't really a bug. You just need to pass the correct parameters. I'll
> > > explain.
> > > 
> > > The error you got is here:
> > > 
> > > imagename: registry.redhat.io/rhceph-beta/rhceph-4-rhel8:latest
> > > Image prepare failed: 401 Client Error: Unauthorized for url:
> > > https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/
> > > auth?service=docker-registry&scope=repository%3Arhceph-beta%2Frhceph-4-
> > > rhel8%3Apull
> > > 
> > > This is because you need a Red Hat account to access our container images.
> > > To use your Red Hat account to access the Red Hat images please see this
> > > docbug
> > > 
> > >  https://bugzilla.redhat.com/show_bug.cgi?id=1723969
> > > 
> > 
> > Can we provide a pre-deployment validation that ensures ceph credentials
> > were provided and do not start the deployment 
> 
> I expect most customers will store the container registry on their
> undercloud by running a command like the following before they run
> 'openstack overcloud deploy':
> 
>  sudo openstack tripleo container image prepare -e containers.yaml
> --output-env-file containers-env-file.yaml

This is only recommended when modifying container images:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html/transitioning_to_containerized_services/obtaining-container-images#modifying-images-during-preparation

> The above downloads the containers from the external Red Hat registry (using
> credentials in containers.yaml) and sets them up to be served from the
> undercloud. Then the overcloud nodes get the containers from the undercloud,
> provided that the overcloud is deployed with '-e containers-env-file.yaml'.
> 
> When the customer follows this pattern without providing their credentials
> they'll fail the authentication before the overcloud deploy and get the
> feedback early. This the pattern that's recommended recommended in our docs.
> See section 2.1:
> 
>  https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/
> html/transitioning_to_containerized_services/obtaining-container-images

Not really, the docs recommend using:

openstack tripleo container image prepare default \
  --local-push-destination \
  --output-env-file containers-prepare-parameter.yaml

which is the same that we're using in testing and seeing the authentication error late in the deployment process.


> The OSP15 version of the above will be updated with how to use
> ContainerImageRegistryCredentials as per docbug 1723969.
> 
> > when credentials are not
> > present? If the ceph registry now requires authentication then imho we
> > shouldn't start deployment if credentials were not provided.
> 
> They're not exactly ceph credentials, they're Red Hat registry credentials
> and I expect in OSP16 that the OpenStack containers will also be hosted on
> registry.redhat.io which will continue to require authentication.
> 
> This issue isn't related to Ceph integration. It's related to director's
> overall process for obtaining container images and the docs will be updated
> to cover accessing our containers from our new registry. If you still think
> we need additional validations then perhaps this should be moved to an RFE
> so PM can assess it and it can be scheduled for a future feature.

Agreed, I'll change the title for a better description.

*** Bug 1726567 has been marked as a duplicate of this bug. ***