Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1684278

Summary: Run tripleo-container-image-prepare logged to /var/log/tripleo-container-image-prepare.log task fails when using insecure registry
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-commonAssignee: Cédric Jeanneret <cjeanner>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 15.0 (Stein)CC: aschultz, cjeanner, dbecker, emacchi, jcoufal, mburns, morazi, sclewis, slinaber
Target Milestone: betaKeywords: Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-10.4.1-0.20190313060345.1377727.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-21 11:20:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2019-02-28 21:28:44 UTC
Description of problem:
Run tripleo-container-image-prepare logged to /var/log/tripleo-container-image-prepare.log task fails when using insecure registry without providing a comprehensible reason of the error:

Starting new HTTPS connection (1): 192.168.24.1:8787
Using config files: ['/tmp/tmphg533y4n']
container_images JSON: [{'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cron:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-haproxy:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-heat-api-cfn:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-heat-api:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-heat-engine:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-ironic-api:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-ironic-conductor:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-ironic-inspector:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-ironic-pxe:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-ironic-neutron-agent:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-iscsid:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-keepalived:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-keystone:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-mariadb:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-memcached:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-mistral-api:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-mistral-engine:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-mistral-executor:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-mistral-event-engine:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-neutron-dhcp-agent:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-neutron-l3-agent:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-neutron-openvswitch-agent:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-neutron-server:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-nova-api:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-nova-compute-ironic:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-nova-conductor:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-nova-placement-api:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-nova-scheduler:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-qdrouterd:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-rabbitmq:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-swift-account:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-swift-container:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-swift-object:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-swift-proxy-server:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-tripleo-ui:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-zaqar-wsgi:20190226.1', 'push_destination': '192.168.24.1:8787'}, {'image_source': 'kolla', 'imagename': 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-tempest:20190226.1', 'push_destination': '192.168.24.1:8787'}]
Starting new HTTPS connection (1): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
imagename: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-tempest:20190226.1
Starting new HTTP connection (1): 192.168.24.1:8787
http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
http://192.168.24.1:8787/v2/ status code 200
http://192.168.24.1:8787 "POST /v2/rhosp15/openstack-tempest/blobs/uploads/ HTTP/1.1" 404 241
Starting new HTTP connection (1): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 "GET /v2/ HTTP/1.1" 200 2
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/v2/ status code 200
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 "GET /v2/rhosp15/openstack-tempest/manifests/20190226.1 HTTP/1.1" 302 581
Starting new HTTPS connection (1): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:443
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 "GET /v2/rhosp15/openstack-tempest/manifests/20190226.1 HTTP/1.1" 302 581
Starting new HTTPS connection (2): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:443
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 "GET /v2/rhosp15/openstack-tempest/manifests/20190226.1 HTTP/1.1" 302 581
Starting new HTTPS connection (3): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:443
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 "GET /v2/rhosp15/openstack-tempest/manifests/20190226.1 HTTP/1.1" 302 581
Starting new HTTPS connection (4): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:443
Resetting dropped connection: brew-pulp-docker01.web.prod.ext.phx2.redhat.com
http://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 "GET /v2/rhosp15/openstack-tempest/manifests/20190226.1 HTTP/1.1" 302 581
Starting new HTTPS connection (5): brew-pulp-docker01.web.prod.ext.phx2.redhat.com:443

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-10.3.1-0.20190227020336.7adc623.el8ost.noarch
python3-tripleo-common-10.4.1-0.20190227000340.1d415e6.el8ost.noarch
openstack-tripleo-common-10.4.1-0.20190227000340.1d415e6.el8ost.noarch
openstack-tripleo-common-containers-10.4.1-0.20190227000340.1d415e6.el8ost.noarc

Comment 1 Cédric Jeanneret 2019-03-04 14:51:17 UTC
Hey,

So for the TLS issue, the mentioned patch will be enough, since a downstream patch will add the internal registry in the NO_VERIFY_REGISTRIES set. So it's "work in progress", patch is in upstream gate right now.

Comment 2 Cédric Jeanneret 2019-03-05 11:36:07 UTC
Linked internal gerrit for the addition of the internal registry.

Comment 3 Cédric Jeanneret 2019-03-05 13:18:31 UTC
So apparently there were some more work to do in the image_uploader.py ... the scoping is this file is terrible :(.

Comment 4 Cédric Jeanneret 2019-03-07 11:20:43 UTC
Hello Marius,

Small update on this:
- this patch is needed as it addresses the last issues: https://review.openstack.org/640941
- as discussed, the particular case of the internal registry needs a particular solution - in this case, infrared should install the IT CA on the undercloud in order to avoid this issue. Care to point me where to push a patch for that?

Cheers,

C.

Comment 5 Cédric Jeanneret 2019-03-08 06:36:41 UTC
Hello Marius,

All is merged - do we have anything to do for downstream?

Cheers,

C.

Comment 16 errata-xmlrpc 2019-09-21 11:20:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811