Bug 1485400

Summary: overcloud container image upload fails due to missing containers
Product: Red Hat OpenStack Reporter: Dan Prince <dprince>
Component: openstack-tripleo-commonAssignee: Dan Prince <dprince>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: adhingra, jjoyce, jschluet, m.andre, mburns, ohochman, sbaker, slinaber
Target Milestone: betaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-7.6.3-0.20171010234828.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:58:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Prince 2017-08-25 14:58:34 UTC
Description of problem:

When running 'openstack overcloud container image upload' it will fail when using a file generated via the new 'openstack overcloud container image prepare' command. This is because


Version-Release number of selected component (if applicable):

[root@undercloud-0 ~]# rpm -qa | grep openstack-tripleo-common
openstack-tripleo-common-containers-7.4.1-0.20170807001945.8c46306.el7ost.noarch
openstack-tripleo-common-7.4.1-0.20170807001945.8c46306.el7ost.noarch

[root@undercloud-0 ~]# rpm -qa | grep python-tripleoclient
python-tripleoclient-7.2.1-0.20170807222309.a731597.el7ost.noarch


How reproducible:

always

Steps to Reproduce:
1. openstack overcloud container image prepare --namespace=rhosp12 --env-file=rhos12.yaml --prefix=openstack- --suffix=-docker --tag=2017-08-18.2 --push-destination 172.19.0.3:8787 --images-file images_file.yaml --pull-source docker-registry.engineering.redhat.com

2. openstack overcloud container image upload --debug --config-file=images_file.yaml

Actual results:

A few images get uploaded and it fails on barbican (a container we do not yet support):

"POST /v1.24/images/create?tag=2017-08-18.2&fromImage=docker-registry.engineering.redhat.com%2Frhosp12%2Fopenstack-barbican-api-docker HTTP/1.1" 200 None
['{"status":"Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker ... "}\r\n', '{"status":"Pulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker"}\r\n', '{"errorDetail":{"message":"Error: image rhosp12/openstack-barbican-api-docker:2017-08-18.2 not found"},"error":"Error: image rhosp12/openstack-barbican-api-docker:2017-08-18.2 not found"}\r\n']
"POST /v1.24/images/docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker:2017-08-18.2/tag?repo=172.19.0.3%3A8787%2Frhosp12%2Fopenstack-barbican-api-docker&tag=2017-08-18.2&force=1 HTTP/1.1" 404 116
404 Client Error: Not Found ("{"message":"no such id: docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker:2017-08-18.2"}")
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 58, in take_action
    uploader.upload()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 64, in upload
    uploader.upload_image(image_name, pull_source, push_destination)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 115, in upload_image
    tag=tag, force=True)
  File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 245, in tag
    self._raise_for_status(res)
  File "/usr/lib/python2.7/site-packages/docker/client.py", line 173, in _raise_for_status
    raise errors.NotFound(e, response, explanation=explanation)
NotFound: 404 Client Error: Not Found ("{"message":"no such id: docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker:2017-08-18.2"}")
clean_up UploadImage: 404 Client Error: Not Found ("{"message":"no such id: docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker:2017-08-18.2"}")
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 58, in take_action
    uploader.upload()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 64, in upload
    uploader.upload_image(image_name, pull_source, push_destination)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 115, in upload_image
    tag=tag, force=True)
  File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 245, in tag
    self._raise_for_status(res)
  File "/usr/lib/python2.7/site-packages/docker/client.py", line 173, in _raise_for_status
    raise errors.NotFound(e, response, explanation=explanation)
NotFound: 404 Client Error: Not Found ("{"message":"no such id: docker-registry.engineering.redhat.com/rhosp12/openstack-barbican-api-docker:2017-08-18.2"}")

Comment 1 Dan Prince 2017-08-25 15:06:59 UTC
Looks like the container images for congress, barbican, and tacker are all missing and as such the 'container images upload' command files. In the example I had above I had to manually update the generated images_files.yaml file like this to get it to pass.

[root@undercloud-0 ~]# diff images_file.yaml.orig images_file.yaml
14,16c14,16
< - imagename: rhosp12/openstack-barbican-api-docker:2017-08-18.2
<   pull_source: docker-registry.engineering.redhat.com
<   push_destination: 172.19.0.3:8787
---
> #- imagename: rhosp12/openstack-barbican-api-docker:2017-08-18.2
>   #pull_source: docker-registry.engineering.redhat.com
>   #push_destination: 172.19.0.3:8787
44,46c44,46
< - imagename: rhosp12/openstack-congress-api-docker:2017-08-18.2
<   pull_source: docker-registry.engineering.redhat.com
<   push_destination: 172.19.0.3:8787
---
> #- imagename: rhosp12/openstack-congress-api-docker:2017-08-18.2
>   #pull_source: docker-registry.engineering.redhat.com
>   #push_destination: 172.19.0.3:8787
245,247c245,247
< - imagename: rhosp12/openstack-tacker-docker:2017-08-18.2
<   pull_source: docker-registry.engineering.redhat.com
<   push_destination: 172.19.0.3:8787
---
> #- imagename: rhosp12/openstack-tacker-docker:2017-08-18.2
>   #pull_source: docker-registry.engineering.redhat.com
>   #push_destination: 172.19.0.3:8787

Comment 2 Dan Prince 2017-08-25 15:09:02 UTC
Jon: could you confirm that we aren't releasing barbican, tacker, and congress containers at this time. And if not should we perhaps put in a patch to exclude these from our 'image prepare' templates?

Comment 4 Steve Baker 2017-08-27 21:52:15 UTC
The prepare command has the following options for filtering entries:

  --exclude <regex>     Pattern to match against resulting imagename entries
                        to exclude from the final output. Can be specified
                        multiple times.
  --service-environment-file <file path>, -e <file path>
                        Environment files specifying which services are
                        containerized. Entries will be filtered to only
                        contain images used by containerized services. (Can be
                        specified more than once.)


But also I think there is a good case for downstream carrying a patch against tripleo-common/container-images/overcloud_containers.yaml.j2 to remove entries for images which downstream does not support.

Comment 5 Jon Schlueter 2017-08-28 03:28:44 UTC
(In reply to Dan Prince from comment #2)
> Jon: could you confirm that we aren't releasing barbican, tacker, and
> congress containers at this time. And if not should we perhaps put in a
> patch to exclude these from our 'image prepare' templates?

That is  correct for OSP 12 we will not be packaging and shipping taker, barbican, and congress container images. Also to follow up on comment #4 yes this would probably be a good patch to carry for OSP 12.

Comment 8 Omri Hochman 2017-10-26 20:03:26 UTC
unable to reproduce with : OpenStack-12.0-RHEL-7 Puddle: 2017-10-24.3

Comment 15 errata-xmlrpc 2017-12-13 21:58:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462