Bug 1455683
| Summary: | rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1" | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Jiri Stransky <jstransk> |
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 12.0 (Pike) | CC: | aschultz, dbecker, jcoufal, jjoyce, jschluet, jstransk, m.andre, mburns, morazi, ohochman, rhel-osp-director-maint, sasha, slinaber, tvignaud |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-7.0.3-3.el7ost openstack-puppet-modules-11.0.0-0.20170828113154.el7ost instack-undercloud-7.4.3-2.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 21:29:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Reproduced. Are there any recent occurrences of this bug? We've implemented the w/a everywhere - re-running the upload command. In order to check it, will need to remove the w/a. Was there anything done to fix it? There wasn't any fix, i still think the current information we have available suggests it's probably an internal infrastructure issue. When using the upstream registries, i never hit this nor did i come across anyone mention this. (In reply to Jiri Stransky from comment #4) > Are there any recent occurrences of this bug? Issue still occurs .. need to re-run the command and it usually works the second time . (In reply to Omri Hochman from comment #7) > (In reply to Jiri Stransky from comment #4) > > Are there any recent occurrences of this bug? > > Issue still occurs .. need to re-run the command and it usually works the > second time . reproduced on Bare-Metal environment. Martin and Dan added retry logic into the places that download containers: https://github.com/openstack/tripleo-heat-templates/commit/d3b3361a76c2e8b188fa8e586d9fb7f3c60bb66f https://github.com/openstack/tripleo-common/commit/2ff0842fcb27af43903d1738bff26bec558c966a Specifically the latter might be relevant here. I still haven't hit the problem myself so i can't test if it fixes this very bug, but i think the patch is worth backporting either way. Proposed a stable/pike backport at https://review.openstack.org/517305 Verified: Environment: openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch instack-undercloud-7.4.3-2.el7ost.noarch The reported issue doesn't reproduce. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1" Environment: openstack-tripleo-heat-templates-7.0.0-0.20170512193554.el7ost.noarch instack-undercloud-7.0.0-0.20170503001109.el7ost.noarch openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch Steps to reproduce: Run: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml Result: 17:35:21 TASK [containerized_deployment : openstack overcloud container image upload] core_puddle_version debug.yaml deploy_info.sh instackenv.json roles ss stackrc templates undercloud.conf undercloud_deploy.sh undercloud_install.log undercloud-passwords.conf upgrade 17:37:16 fatal: [undercloud-0]: FAILED! => {"changed": true, "cmd": "source /home/stack/stackrc openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml", "delta": "0:01:54.671984", "end": "2017-05-25 13:37:16.976632", "failed": true, "rc": 1, "start": "2017-05-25 13:35:22.304648", "stderr": "START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/templates/container_images.yaml'] command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False) Using config files: [u'/home/stack/templates/container_images.yaml'] imagename: rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6 /usr/lib/python2.7/site-packages/docker/api/image.py:153: DeprecationWarning: The `insecure_registry` argument to pull() is deprecated and non-functional. Please remove it. DeprecationWarning /usr/lib/python2.7/site-packages/docker/api/image.py:192: DeprecationWarning: The `insecure_registry` argument to push() is deprecated and non-functional. Please remove it. DeprecationWarning Completed upload for docker image rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6 imagename: rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6 imagename: rhosp12/openstack-base-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-base-docker:2017-05-16.6 imagename: rhosp12/openstack-heat-api-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-heat-api-docker:2017-05-16.6 imagename: rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6 imagename: rhosp12/openstack-mistral-engine-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-mistral-engine-docker:2017-05-16.6 imagename: rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6 imagename: rhosp12/openstack-heat-engine-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-heat-engine-docker:2017-05-16.6 imagename: rhosp12/openstack-keystone-ssh-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-keystone-ssh-docker:2017-05-16.6 imagename: rhosp12/openstack-panko-base-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-panko-base-docker:2017-05-16.6 imagename: rhosp12/openstack-redis-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-redis-docker:2017-05-16.6 imagename: rhosp12/openstack-aodh-notifier-docker:2017-05-16.6 Completed upload for docker image rhosp12/openstack-aodh-notifier-docker:2017-05-16.6 imagename: rhosp12/openstack-glance-api-docker:2017-05-16.6 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) END return value: 1", "stderr_lines": ["START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/templates/container_images.yaml']", "command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)", "Using config files: [u'/home/stack/templates/container_images.yaml']", "imagename: rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6", "/usr/lib/python2.7/site-packages/docker/api/image.py:153: DeprecationWarning: The `insecure_registry` argument to pull() is deprecated and non-functional. Please remove it.", " DeprecationWarning", "/usr/lib/python2.7/site-packages/docker/api/image.py:192: DeprecationWarning: The `insecure_registry` argument to push() is deprecated and non-functional. Please remove it.", " DeprecationWarning", "Completed upload for docker image rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6", "imagename: rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6", "imagename: rhosp12/openstack-base-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-base-docker:2017-05-16.6", "imagename: rhosp12/openstack-heat-api-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-heat-api-docker:2017-05-16.6", "imagename: rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6", "imagename: rhosp12/openstack-mistral-engine-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-mistral-engine-docker:2017-05-16.6", "imagename: rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6", "imagename: rhosp12/openstack-heat-engine-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-heat-engine-docker:2017-05-16.6", "imagename: rhosp12/openstack-keystone-ssh-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-keystone-ssh-docker:2017-05-16.6", "imagename: rhosp12/openstack-panko-base-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-panko-base-docker:2017-05-16.6", "imagename: rhosp12/openstack-redis-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-redis-docker:2017-05-16.6", "imagename: rhosp12/openstack-aodh-notifier-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-aodh-notifier-docker:2017-05-16.6", "imagename: rhosp12/openstack-glance-api-docker:2017-05-16.6", "('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"], "stdout": "", "stdout_lines": []} Removing the images and re-running the command works fine.