Bug 1456986 - [OSP12][openstack containers]: Errors while pulling images from remote registry causes overcloud deployment to fail.
Summary: [OSP12][openstack containers]: Errors while pulling images from remote regis...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers
Version: 12.0 (Pike)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: 12.0 (Pike)
Assignee: Jiri Stransky
QA Contact: Alexander Chuzhoy
Andrew Burden
URL:
Whiteboard:
: 1483756 (view as bug list)
Depends On:
Blocks: 1434060
TreeView+ depends on / blocked
 
Reported: 2017-05-30 21:19 UTC by Omri Hochman
Modified: 2017-12-13 19:14 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 19:14:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1715136 0 None None None 2017-09-27 07:12:14 UTC
OpenStack gerrit 498139 0 'None' MERGED Add DockerPuppetProcessCount defaults to 3 2021-01-18 08:59:39 UTC
OpenStack gerrit 505681 0 'None' MERGED Add retry loop for docker pull 2021-01-18 08:59:39 UTC
Red Hat Product Errata RHEA-2017:3457 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Containers Enhancement Advisory 2017-12-14 04:45:51 UTC

Description Omri Hochman 2017-05-30 21:19:57 UTC
[OSP12][openstack containers]:  Errors while pulling images from remote registry   causes overcloud deployment to fail. 


Environment: 
------------
openstack-tripleo-image-elements-6.0.0-2.el7ost.noarch
openstack-tripleo-puppet-elements-6.0.0-3.el7ost.noarch
openstack-tripleo-ui-3.1.0-9.el7ost.noarch
openstack-tripleo-validations-5.5.0-1.el7ost.noarch
python-tripleoclient-6.1.0-6.el7ost.noarch
openstack-tripleo-heat-templates-6.0.0-12.el7ost.noarch
puppet-tripleo-6.3.0-12.el7ost.noarch
openstack-tripleo-common-6.0.0-8.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch


Steps: 
------
Attempt to deploy osp12 with containers :
 1 controller , 1 compute 
overcloud containers - IMAGE_TAG:2017-05-16.6



(undercloud) [root@undercloud-0 ~]# heat  resource-list overcloud -n 5 | grep -v COMPLETE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead

+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                                      | physical_resource_id                                                                                                                                                                 | resource_type                                                                                                          | resource_status | updated_time         | stack_name                                                                                                                                               |
+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| AllNodesDeploySteps                                | b65f2071-8041-4b23-9635-32f0f6df0e2d                                                                                                                                                 | OS::TripleO::PostDeploySteps                                                                                           | CREATE_FAILED   | 2017-05-30T19:34:04Z | overcloud                                                                                                                                                |
| ControllerGenerateConfigDeployment                 | 10548eda-bade-473a-b8e8-57803401a6b8                                                                                                                                                 | OS::Heat::SoftwareDeploymentGroup                                                                                      | CREATE_FAILED   | 2017-05-30T19:48:18Z | overcloud-AllNodesDeploySteps-apdpa2wcrpdf                                                                                                               |
| 0                                                  | c68436a7-4d79-450a-b8ea-7cd2d2f003cd                                                                                                                                                 | OS::Heat::SoftwareDeployment                                                                                           | CREATE_FAILED   | 2017-05-30T19:52:01Z | overcloud-AllNodesDeploySteps-apdpa2wcrpdf-ControllerGenerateConfigDeployment-owykeuiecr65                                                               |
+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

(undercloud) [root@undercloud-0 ~]# heat deployment-show c68436a7-4d79-450a-b8ea-7cd2d2f003cd 
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "037d8a11-8724-4728-8e47-2346b0e4763e", 
  "config_id": "8a25546a-d2d1-46f4-b7ce-03f16d8c2a87", 
  "output_values": {
    "deploy_stdout": "2017-05-30 19:52:38,891 INFO: Running docker-puppet\n2017-05-30 19:52:38,891 DEBUG: CONFIG: /var/lib/docker-puppet/docker-puppet.json\n2017-05-30 19:52:38,891 DEBUG: config_volume keystone\n2017-05-30 19:52:38,892 DEBUG: puppet_tags keystone_config\n2017-05-30 19:52:38,892 DEBUG: manifest ['Keystone_user', 'Keystone_endpoint', 'Keystone_domain', 'Keystone_tenant', 'Keystone_user_role', 'Keystone_role', 'Keystone_service'].each |String $val| { noop_resource($val) }\ninclude ::tripleo::profile::base::keystone\n\n2017-05-30 19:52:38,892 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-05-16.6\n2017-05-30 19:52:38,892 DEBUG: volumes []\n2017-05-30 19:52:38,892 INFO: Adding new service\n2017-05-30 19:52:38,892 DEBUG: config_volume glance_api\n2017-05-30 19:52:38,892 DEBUG: puppet_tags glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config\n2017-05-30 19:52:38,892 DEBUG: manifest include ::tripleo::profile::base::glance::api\n\n2017-05-30 19:52:38,893 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker:2017-05-16.6\n2017-05-30 19:52:38,893 DEBUG: volumes []\n2017-05-30 19:52:38,893 INFO: Adding new service\n2017-05-30 19:52:38,893 DEBUG: config_volume heat_api\n2017-05-30 19:52:38,893 DEBUG: puppet_tags heat_config,file,concat,file_line\n2017-05-30 19:52:38,893 DEBUG: manifest 


...
...
...
...

registry.engineering.redhat.com/rhosp12/openstack-nova-base-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-nova-base-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-nova-base-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,264 ERROR: Failed running docker-puppet.py for nova\n2017-05-30 19:52:39,264 INFO: Removing container: docker-puppet-nova\n2017-05-30 19:52:39,265 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-glance-api-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,265 ERROR: Failed running docker-puppet.py for glance_api\n2017-05-30 19:52:39,266 INFO: Removing container: docker-puppet-glance_api\n2017-05-30 19:52:39,279 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-heat-engine-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,280 ERROR: Failed running docker-puppet.py for heat_api_cfn\n2017-05-30 19:52:39,280 INFO: Removing container: docker-puppet-heat_api_cfn\n2017-05-30 19:52:39,281 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-memcached-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-memcached-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-memcached-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-memcached-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,281 ERROR: Failed running docker-puppet.py for memcached\n2017-05-30 19:52:39,282 INFO: Removing container: docker-puppet-memcached\n2017-05-30 19:52:39,286 DEBUG: config_volume keystone\n2017-05-30 19:52:39,287 DEBUG: puppet_tags file,file_line,concat,augeas,keystone_config\n2017-05-30 19:52:39,287 DEBUG: manifest ['Keystone_user', 'Keystone_endpoint', 'Keystone_domain', 'Keystone_tenant', 'Keystone_user_role', 'Keystone_role', 'Keystone_service'].each |String $val| { noop_resource($val) }\ninclude ::tripleo::profile::base::keystone\n\n2017-05-30 19:52:39,287 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-05-16.6\n2017-05-30 19:52:39,287 DEBUG: volumes []\n2017-05-30 19:52:39,287 INFO: Removing container: docker-puppet-keystone\n2017-05-30 19:52:39,288 DEBUG: config_volume mysql\n2017-05-30 19:52:39,288 DEBUG: puppet_tags file,file_line,concat,augeas,file\n2017-05-30 19:52:39,288 DEBUG: manifest ['Mysql_datadir', 'Mysql_user', 'Mysql_database', 'Mysql_grant', 'Mysql_plugin'].each |String $val| { noop_resource($val) }\ninclude ::tripleo::profile::base::database::mysql\n\n2017-05-30 19:52:39,288 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-05-16.6\n2017-05-30 19:52:39,288 DEBUG: volumes []\n2017-05-30 19:52:39,289 INFO: Removing container: docker-puppet-mysql\n2017-05-30 19:52:39,300 DEBUG: config_volume panko\n2017-05-30 19:52:39,300 DEBUG: puppet_tags file,file_line,concat,augeas,panko_api_paste_ini,panko_config\n2017-05-30 19:52:39,300 DEBUG: manifest include tripleo::profile::base::panko::api\n\n2017-05-30 19:52:39,300 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-05-16.6\n2017-05-30 19:52:39,300 DEBUG: volumes []\n2017-05-30 19:52:39,301 INFO: Removing container: docker-puppet-panko\n2017-05-30 19:52:39,302 DEBUG: config_volume gnocchi\n2017-05-30 19:52:39,302 DEBUG: puppet_tags file,file_line,concat,augeas,gnocchi_api_paste_ini,gnocchi_config,gnocchi_config,gnocchi_config\n2017-05-30 19:52:39,302 DEBUG: manifest include ::tripleo::profile::base::gnocchi::api\n\ninclude ::tripleo::profile::base::gnocchi::metricd\n\ninclude ::tripleo::profile::base::gnocchi::statsd\n\n2017-05-30 19:52:39,302 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-05-16.6\n2017-05-30 19:52:39,303 DEBUG: volumes []\n2017-05-30 19:52:39,303 INFO: Removing container: docker-puppet-gnocchi\n2017-05-30 19:52:39,309 INFO: Pulling image: docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-05-16.6\n2017-05-30 19:52:39,319 INFO: Pulling image: docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-05-16.6\n2017-05-30 19:52:39,319 INFO: Pulling image: docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-05-16.6\n2017-05-30 19:52:39,322 INFO: Pulling image: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-05-16.6\n2017-05-30 19:52:39,345 DEBUG: Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker\n\n2017-05-30 19:52:39,345 DEBUG: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-keystone-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving\n\n2017-05-30 19:52:39,348 DEBUG: NET_HOST enabled\n2017-05-30 19:52:39,348 DEBUG: Running docker command: /usr/bin/docker run --user root --name docker-puppet-keystone --env PUPPET_TAGS=file,file_line,concat,augeas,keystone_config --env NAME=keystone --env HOSTNAME=overcloud-controller-0 --env NO_ARCHIVE= --env STEP=6 --volume /tmp/tmpt76Ixe:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --volume /var/lib/config-data/:/var/lib/config-data/:rw --volume tripleo_logs:/var/log/tripleo/ --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:rw --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-05-16.6\n2017-05-30 19:52:39,348 DEBUG: Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker\n\n2017-05-30 19:52:39,348 DEBUG: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-panko-api-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving\n\n2017-05-30 19:52:39,349 DEBUG: Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker\n\n2017-05-30 19:52:39,349 DEBUG: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-mariadb-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving\n\n2017-05-30 19:52:39,350 DEBUG: Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker\n\n2017-05-30 19:52:39,350 DEBUG: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-gnocchi-api-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving\n\n2017-05-30 19:52:39,351 DEBUG: NET_HOST enabled\n2017-05-30 19:52:39,351 DEBUG: Running docker command: /usr/bin/docker run --user root --name docker-puppet-mysql --env PUPPET_TAGS=file,file_line,concat,augeas,file --env NAME=mysql --env HOSTNAME=overcloud-controller-0 --env NO_ARCHIVE= --env STEP=6 --volume /tmp/tmpL2ASzz:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --volume /var/lib/config-data/:/var/lib/config-data/:rw --volume tripleo_logs:/var/log/tripleo/ --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:rw --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-05-16.6\n2017-05-30 19:52:39,352 DEBUG: NET_HOST enabled\n2017-05-30 19:52:39,352 DEBUG: Running docker command: /usr/bin/docker run --user root --name docker-puppet-panko --env PUPPET_TAGS=file,file_line,concat,augeas,panko_api_paste_ini,panko_config --env NAME=panko --env HOSTNAME=overcloud-controller-0 --env NO_ARCHIVE= --env STEP=6 --volume /tmp/tmp_ONR8x:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --volume /var/lib/config-data/:/var/lib/config-data/:rw --volume tripleo_logs:/var/log/tripleo/ --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:rw --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-05-16.6\n2017-05-30 19:52:39,353 DEBUG: NET_HOST enabled\n2017-05-30 19:52:39,353 DEBUG: Running docker command: /usr/bin/docker run --user root --name docker-puppet-gnocchi --env PUPPET_TAGS=file,file_line,concat,augeas,gnocchi_api_paste_ini,gnocchi_config,gnocchi_config,gnocchi_config --env NAME=gnocchi --env HOSTNAME=overcloud-controller-0 --env NO_ARCHIVE= --env STEP=6 --volume /tmp/tmp5hDn2r:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --volume /var/lib/config-data/:/var/lib/config-data/:rw --volume tripleo_logs:/var/log/tripleo/ --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:rw --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-05-16.6\n2017-05-30 19:52:39,382 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-mariadb-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,383 ERROR: Failed running docker-puppet.py for mysql\n2017-05-30 19:52:39,383 INFO: Removing container: docker-puppet-mysql\n2017-05-30 19:52:39,387 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-gnocchi-api-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,387 ERROR: Failed running docker-puppet.py for gnocchi\n2017-05-30 19:52:39,387 INFO: Removing container: docker-puppet-gnocchi\n2017-05-30 19:52:39,388 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-keystone-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,388 ERROR: Failed running docker-puppet.py for keystone\n2017-05-30 19:52:39,388 INFO: Removing container: docker-puppet-keystone\n2017-05-30 19:52:39,391 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-panko-api-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,391 ERROR: Failed running docker-puppet.py for panko\n2017-05-30 19:52:39,391 INFO: Removing container: docker-puppet-panko\n2017-05-30 19:52:39,403 DEBUG: config_volume neutron\n2017-05-30 19:52:39,403 DEBUG: puppet_tags file,file_line,concat,augeas,neutron_config,neutron_dhcp_agent_config,neutron_config,neutron_l3_agent_config,neutron_config,neutron_api_config,neutron_config,neutron_agent_ovs,neutron_plugin_ml2\n2017-05-30 19:52:39,403 DEBUG: manifest include tripleo::profile::base::neutron::dhcp\n\ninclude tripleo::profile::base::neutron::l3\n\ninclude tripleo::profile::base::neutron::server\n\ninclude ::tripleo::profile::base::neutron::plugins::ml2\n\ninclude ::tripleo::profile::base::neutron::ovs\n\n2017-05-30 19:52:39,403 DEBUG: config_image docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker:2017-05-16.6\n2017-05-30 19:52:39,403 DEBUG: volumes []\n2017-05-30 19:52:39,403 INFO: Removing container: docker-puppet-neutron\n2017-05-30 19:52:39,418 INFO: Pulling image: docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker:2017-05-16.6\n2017-05-30 19:52:39,446 DEBUG: Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker\n\n2017-05-30 19:52:39,446 DEBUG: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-neutron-openvswitch-agent-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving\n\n2017-05-30 19:52:39,449 DEBUG: NET_HOST enabled\n2017-05-30 19:52:39,449 DEBUG: Running docker command: /usr/bin/docker run --user root --name docker-puppet-neutron --env PUPPET_TAGS=file,file_line,concat,augeas,neutron_config,neutron_dhcp_agent_config,neutron_config,neutron_l3_agent_config,neutron_config,neutron_api_config,neutron_config,neutron_agent_ovs,neutron_plugin_ml2 --env NAME=neutron --env HOSTNAME=overcloud-controller-0 --env NO_ARCHIVE= --env STEP=6 --volume /tmp/tmpahS6GV:/etc/config.pp:ro --volume /etc/puppet/:/tmp/puppet-etc/:ro --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro --volume /var/lib/config-data/:/var/lib/config-data/:rw --volume tripleo_logs:/var/log/tripleo/ --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:rw --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker:2017-05-16.6\n2017-05-30 19:52:39,478 DEBUG: Unable to find image 'docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker:2017-05-16.6' locally\nTrying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker ... \nPulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-neutron-openvswitch-agent-docker\n/usr/bin/docker-current: Error while pulling image: Get http://docker-registry.engineering.redhat.com/v1/repositories/rhosp12/openstack-neutron-openvswitch-agent-docker/images: dial tcp: lookup docker-registry.engineering.redhat.com on 192.168.24.5:53: server misbehaving.\nSee '/usr/bin/docker-current run --help'.\n\n2017-05-30 19:52:39,478 ERROR: Failed running docker-puppet.py for neutron\n2017-05-30 19:52:39,478 INFO: Removing container: docker-puppet-neutron\n2017-05-30 19:52:39,496 ERROR: ERROR configuring nova_placement\n2017-05-30 19:52:39,496 ERROR: ERROR configuring swift\n2017-05-30 19:52:39,497 ERROR: ERROR configuring mongodb\n2017-05-30 19:52:39,497 ERROR: ERROR configuring heat_api\n2017-05-30 19:52:39,497 ERROR: ERROR configuring redis\n2017-05-30 19:52:39,497 ERROR: ERROR configuring rabbitmq\n2017-05-30 19:52:39,497 ERROR: ERROR configuring aodh\n2017-05-30 19:52:39,497 ERROR: ERROR configuring heat\n2017-05-30 19:52:39,497 ERROR: ERROR configuring glance_api\n2017-05-30 19:52:39,497 ERROR: ERROR configuring keystone\n2017-05-30 19:52:39,498 ERROR: ERROR configuring nova\n2017-05-30 19:52:39,498 ERROR: ERROR configuring mysql\n2017-05-30 19:52:39,498 ERROR: ERROR configuring heat_api_cfn\n2017-05-30 19:52:39,498 ERROR: ERROR configuring gnocchi\n2017-05-30 19:52:39,498 ERROR: ERROR configuring memcached\n2017-05-30 19:52:39,498 ERROR: ERROR configuring panko\n2017-05-30 19:52:39,498 ERROR: ERROR configuring neutron\n", 
    "deploy_stderr": "", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2017-05-30T19:52:01Z", 
  "updated_time": "2017-05-30T19:52:42Z", 
  "input_values": {
    "NET_HOST": "true"
  }, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "c68436a7-4d79-450a-b8ea-7cd2d2f003cd"
}
(undercloud) [root@undercloud-0 ~]#

Comment 1 Red Hat Bugzilla Rules Engine 2017-05-30 21:20:02 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 2 Omri Hochman 2017-05-30 21:30:15 UTC
We should check if the issue is related with Bz: https://bugzilla.redhat.com/show_bug.cgi?id=1455683 


Another question would be: 
--------------------------
why do we need to have the remote registry configured during overcloud deployment, if the images are already downloaded and pushed on the local registry on the undercloud ? 


local : 192.168.24.1:8787
remote: docker-registry.engineering.redhat.com

Comment 3 Jiri Stransky 2017-05-31 08:12:58 UTC
(In reply to Omri Hochman from comment #2)
> why do we need to have the remote registry configured during overcloud
> deployment, if the images are already downloaded and pushed on the local
> registry on the undercloud ? 

This all depends on the DockerNamespace parameter value you provided to the deployment. From the logs above it looks like you pointed DockerNamespace at docker-registry.engineering.redhat.com directly (i guess as an attempt to work around bug 1455683 ?). In this case it would download the images from that registry instead of using the one on undercloud.

Comment 4 Omri Hochman 2017-06-05 19:40:24 UTC
If we would go with a solution (such as remote SAT )  or anything else , that does not include to build the local registry on the undercloud , then we would need to check again  -  If during overcloud deployment we're getting connection time-outs that breaks deployment ?  In that case it would be a blocker.

Comment 7 Bogdan Dobrelya 2017-06-22 12:03:54 UTC
Omri, I believe intermittent download issues can not be treated as blockers. The same type of issues may be faced with installing packages. How should these situations be handled? Is automatic retry steps required, like "yum clean expire-cache && yum history redo last" for the latter example? Would manual recovery steps make it a blocker as well?

Comment 8 Jiri Stransky 2017-07-25 14:46:09 UTC
Are there any recent occurrences of this bug?

Comment 9 Dan Prince 2017-08-26 03:09:14 UTC
I'm hitting this as well now. One solution I found was to set docker-puppet.py's PROCESS_COUNT env variable to 3. This matches docker daemon's default pull count as well and might be a safer default.

I've proposed this in the upstream patch here:

https://bugs.launchpad.net/tripleo/+bug/1713188

Comment 10 Omri Hochman 2017-08-28 19:25:44 UTC

(In reply to Jiri Stransky from comment #8)
> Are there any recent occurrences of this bug?


According to Comment#9 seems that Dan just reproduced it on latest version , and then, changing the settings fixed the issue.

Comment 11 Dan Prince 2017-09-21 13:37:57 UTC
In addition to the PROCESS_COUNT patch above there is also a docker pull retry we added in

commit 9a1015581d59a7a38e1bdb2ff97da1161123e05c
Author: Dan Prince <dprince>
Date:   Thu Sep 7 16:48:28 2017 -0400

    Add a docker pull retry to docker-puppet.py
    
    Co-Authored-By: Ian Main <imain>
    
    Change-Id: Iad6d38690340f4a064a4527c58ed439d91fa5188
    Closes-bug: #1715136
    (cherry picked from commit d3b3361a76c2e8b188fa8e586d9fb7f3c60bb66f)

Both of these are in the latest puddles.

Curios to see if anyone from QE is still hitting this.

Comment 12 Bob Fournier 2017-09-26 20:28:25 UTC
*** Bug 1483756 has been marked as a duplicate of this bug. ***

Comment 13 Martin André 2017-09-27 07:15:39 UTC
And I've also added the same retry loop to the docker pull from the 'overcloud container image upload' command in https://review.openstack.org/#/c/505681/ to accommodate with occasional I/O failures.

Comment 16 Alexander Chuzhoy 2017-10-27 01:05:05 UTC
Environment:
instack-undercloud-7.4.2-0.20171010064304.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
openstack-tripleo-common-containers-7.6.3-0.20171022171808.el7ost.noarch
openstack-tripleo-common-7.6.3-0.20171022171808.el7ost.noarch


Able to deploy successfully with remote registry

Comment 20 errata-xmlrpc 2017-12-13 19:14:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3457


Note You need to log in before you can comment on or make changes to this bug.