Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1455683 - rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"
rhosp-director: openstack overcloud container image upload --verbose --config...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity high
: beta
: 12.0 (Pike)
Assigned To: Jiri Stransky
Alexander Chuzhoy
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-05-25 15:34 EDT by Alexander Chuzhoy
Modified: 2018-02-05 14:07 EST (History)
14 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.3-3.el7ost openstack-puppet-modules-11.0.0-0.20170828113154.el7ost instack-undercloud-7.4.3-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:29:42 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 517305 None None None 2017-11-02 09:34 EDT
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-15 20:43:25 EST

  None (edit)
Description Alexander Chuzhoy 2017-05-25 15:34:02 EDT
rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml  fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"


Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170512193554.el7ost.noarch
instack-undercloud-7.0.0-0.20170503001109.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch


Steps to reproduce:
Run:
openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml

Result:
17:35:21 TASK [containerized_deployment : openstack overcloud container image upload] core_puddle_version debug.yaml deploy_info.sh instackenv.json roles ss stackrc templates undercloud.conf undercloud_deploy.sh undercloud_install.log undercloud-passwords.conf upgrade 17:37:16 fatal: [undercloud-0]: FAILED! => {"changed": true, "cmd": "source /home/stack/stackrc
 openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml", "delta": "0:01:54.671984", "end": "2017-05-25 13:37:16.976632", "failed": true, "rc": 1, "start": "2017-05-25 13:35:22.304648", "stderr": "START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/templates/container_images.yaml']
command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)
Using config files: [u'/home/stack/templates/container_images.yaml']
imagename: rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6
/usr/lib/python2.7/site-packages/docker/api/image.py:153: DeprecationWarning: The `insecure_registry` argument to pull() is deprecated and non-functional. Please remove it.
 DeprecationWarning
/usr/lib/python2.7/site-packages/docker/api/image.py:192: DeprecationWarning: The `insecure_registry` argument to push() is deprecated and non-functional. Please remove it.
 DeprecationWarning
Completed upload for docker image rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6
imagename: rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6
imagename: rhosp12/openstack-base-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-base-docker:2017-05-16.6
imagename: rhosp12/openstack-heat-api-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-heat-api-docker:2017-05-16.6
imagename: rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6
imagename: rhosp12/openstack-mistral-engine-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-mistral-engine-docker:2017-05-16.6
imagename: rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6
imagename: rhosp12/openstack-heat-engine-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-heat-engine-docker:2017-05-16.6
imagename: rhosp12/openstack-keystone-ssh-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-keystone-ssh-docker:2017-05-16.6
imagename: rhosp12/openstack-panko-base-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-panko-base-docker:2017-05-16.6
imagename: rhosp12/openstack-redis-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-redis-docker:2017-05-16.6
imagename: rhosp12/openstack-aodh-notifier-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-aodh-notifier-docker:2017-05-16.6
imagename: rhosp12/openstack-glance-api-docker:2017-05-16.6
('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
END return value: 1", "stderr_lines": ["START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/templates/container_images.yaml']", "command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)", "Using config files: [u'/home/stack/templates/container_images.yaml']", "imagename: rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6", "/usr/lib/python2.7/site-packages/docker/api/image.py:153: DeprecationWarning: The `insecure_registry` argument to pull() is deprecated and non-functional. Please remove it.", " DeprecationWarning", "/usr/lib/python2.7/site-packages/docker/api/image.py:192: DeprecationWarning: The `insecure_registry` argument to push() is deprecated and non-functional. Please remove it.", " DeprecationWarning", "Completed upload for docker image rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6", "imagename: rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6", "imagename: rhosp12/openstack-base-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-base-docker:2017-05-16.6", "imagename: rhosp12/openstack-heat-api-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-heat-api-docker:2017-05-16.6", "imagename: rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6", "imagename: rhosp12/openstack-mistral-engine-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-mistral-engine-docker:2017-05-16.6", "imagename: rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6", "imagename: rhosp12/openstack-heat-engine-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-heat-engine-docker:2017-05-16.6", "imagename: rhosp12/openstack-keystone-ssh-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-keystone-ssh-docker:2017-05-16.6", "imagename: rhosp12/openstack-panko-base-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-panko-base-docker:2017-05-16.6", "imagename: rhosp12/openstack-redis-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-redis-docker:2017-05-16.6", "imagename: rhosp12/openstack-aodh-notifier-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-aodh-notifier-docker:2017-05-16.6", "imagename: rhosp12/openstack-glance-api-docker:2017-05-16.6", "('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"], "stdout": "", "stdout_lines": []}



Removing the images and re-running the command works fine.
Comment 1 Alexander Chuzhoy 2017-05-26 16:42:59 EDT
Reproduced.
Comment 4 Jiri Stransky 2017-07-25 10:46:07 EDT
Are there any recent occurrences of this bug?
Comment 5 Alexander Chuzhoy 2017-07-25 10:56:42 EDT
We've implemented the w/a everywhere -  re-running the upload command.
In order to check it, will need to remove the w/a. Was there anything done to fix it?
Comment 6 Jiri Stransky 2017-07-26 07:18:58 EDT
There wasn't any fix, i still think the current information we have available suggests it's probably an internal infrastructure issue. When using the upstream registries, i never hit this nor did i come across anyone mention this.
Comment 7 Omri Hochman 2017-08-10 13:38:38 EDT
(In reply to Jiri Stransky from comment #4)
> Are there any recent occurrences of this bug?

Issue still occurs ..  need to re-run the command and it usually works the second time .
Comment 8 Omri Hochman 2017-08-10 13:55:04 EDT
(In reply to Omri Hochman from comment #7)
> (In reply to Jiri Stransky from comment #4)
> > Are there any recent occurrences of this bug?
> 
> Issue still occurs ..  need to re-run the command and it usually works the
> second time .
 
reproduced on Bare-Metal environment.
Comment 10 Jiri Stransky 2017-11-02 09:34:57 EDT
Martin and Dan added retry logic into the places that download containers:

https://github.com/openstack/tripleo-heat-templates/commit/d3b3361a76c2e8b188fa8e586d9fb7f3c60bb66f

https://github.com/openstack/tripleo-common/commit/2ff0842fcb27af43903d1738bff26bec558c966a

Specifically the latter might be relevant here. I still haven't hit the problem myself so i can't test if it fixes this very bug, but i think the patch is worth backporting either way. Proposed a stable/pike backport at https://review.openstack.org/517305
Comment 12 Alexander Chuzhoy 2017-11-15 12:45:24 EST
Verified:

Environment:
openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
instack-undercloud-7.4.3-2.el7ost.noarch


The reported issue doesn't reproduce.
Comment 16 errata-xmlrpc 2017-12-13 16:29:42 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.