Bug 1455683 – rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1455683 - rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"

Summary:	rhosp-director: openstack overcloud container image upload --verbose --config...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates (Show other bugs)
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified Unspecified

Priority	high Severity high
Target Milestone:	beta
Target Release:	12.0 (Pike)
Assigned To:	Jiri Stransky
QA Contact:	Alexander Chuzhoy
Docs Contact:

URL:
Whiteboard:
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2017-05-25 15:34 EDT by Alexander Chuzhoy
Modified:	2018-02-05 14:07 EST (History)
CC List:	14 users (show)

See Also:
Fixed In Version:	openstack-tripleo-heat-templates-7.0.3-3.el7ost openstack-puppet-modules-11.0.0-0.20170828113154.el7ost instack-undercloud-7.4.3-2.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-12-13 16:29:42 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	517305	None	None	None	2017-11-02 09:34 EDT
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-15 20:43:25 EST

Groups: None (edit)

Description Alexander Chuzhoy 2017-05-25 15:34:02 EDT

rhosp-director: openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml  fails with "'Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"


Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170512193554.el7ost.noarch
instack-undercloud-7.0.0-0.20170503001109.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch


Steps to reproduce:
Run:
openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml

Result:
17:35:21 TASK [containerized_deployment : openstack overcloud container image upload] core_puddle_version debug.yaml deploy_info.sh instackenv.json roles ss stackrc templates undercloud.conf undercloud_deploy.sh undercloud_install.log undercloud-passwords.conf upgrade 17:37:16 fatal: [undercloud-0]: FAILED! => {"changed": true, "cmd": "source /home/stack/stackrc
 openstack overcloud container image upload --verbose --config-file /home/stack/templates/container_images.yaml", "delta": "0:01:54.671984", "end": "2017-05-25 13:37:16.976632", "failed": true, "rc": 1, "start": "2017-05-25 13:35:22.304648", "stderr": "START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/templates/container_images.yaml']
command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)
Using config files: [u'/home/stack/templates/container_images.yaml']
imagename: rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6
/usr/lib/python2.7/site-packages/docker/api/image.py:153: DeprecationWarning: The `insecure_registry` argument to pull() is deprecated and non-functional. Please remove it.
 DeprecationWarning
/usr/lib/python2.7/site-packages/docker/api/image.py:192: DeprecationWarning: The `insecure_registry` argument to push() is deprecated and non-functional. Please remove it.
 DeprecationWarning
Completed upload for docker image rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6
imagename: rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6
imagename: rhosp12/openstack-base-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-base-docker:2017-05-16.6
imagename: rhosp12/openstack-heat-api-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-heat-api-docker:2017-05-16.6
imagename: rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6
imagename: rhosp12/openstack-mistral-engine-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-mistral-engine-docker:2017-05-16.6
imagename: rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6
imagename: rhosp12/openstack-heat-engine-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-heat-engine-docker:2017-05-16.6
imagename: rhosp12/openstack-keystone-ssh-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-keystone-ssh-docker:2017-05-16.6
imagename: rhosp12/openstack-panko-base-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-panko-base-docker:2017-05-16.6
imagename: rhosp12/openstack-redis-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-redis-docker:2017-05-16.6
imagename: rhosp12/openstack-aodh-notifier-docker:2017-05-16.6
Completed upload for docker image rhosp12/openstack-aodh-notifier-docker:2017-05-16.6
imagename: rhosp12/openstack-glance-api-docker:2017-05-16.6
('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
END return value: 1", "stderr_lines": ["START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/home/stack/templates/container_images.yaml']", "command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)", "Using config files: [u'/home/stack/templates/container_images.yaml']", "imagename: rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6", "/usr/lib/python2.7/site-packages/docker/api/image.py:153: DeprecationWarning: The `insecure_registry` argument to pull() is deprecated and non-functional. Please remove it.", " DeprecationWarning", "/usr/lib/python2.7/site-packages/docker/api/image.py:192: DeprecationWarning: The `insecure_registry` argument to push() is deprecated and non-functional. Please remove it.", " DeprecationWarning", "Completed upload for docker image rhosp12/openstack-heat-api-cloudwatch-docker:2017-05-16.6", "imagename: rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-swift-object-expirer-docker:2017-05-16.6", "imagename: rhosp12/openstack-base-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-base-docker:2017-05-16.6", "imagename: rhosp12/openstack-heat-api-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-heat-api-docker:2017-05-16.6", "imagename: rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-nova-compute-ironic-docker:2017-05-16.6", "imagename: rhosp12/openstack-mistral-engine-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-mistral-engine-docker:2017-05-16.6", "imagename: rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-aodh-evaluator-docker:2017-05-16.6", "imagename: rhosp12/openstack-heat-engine-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-heat-engine-docker:2017-05-16.6", "imagename: rhosp12/openstack-keystone-ssh-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-keystone-ssh-docker:2017-05-16.6", "imagename: rhosp12/openstack-panko-base-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-panko-base-docker:2017-05-16.6", "imagename: rhosp12/openstack-redis-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-redis-docker:2017-05-16.6", "imagename: rhosp12/openstack-aodh-notifier-docker:2017-05-16.6", "Completed upload for docker image rhosp12/openstack-aodh-notifier-docker:2017-05-16.6", "imagename: rhosp12/openstack-glance-api-docker:2017-05-16.6", "('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))", "END return value: 1"], "stdout": "", "stdout_lines": []}



Removing the images and re-running the command works fine.

Comment 1 Alexander Chuzhoy 2017-05-26 16:42:59 EDT

Reproduced.

Comment 4 Jiri Stransky 2017-07-25 10:46:07 EDT

Are there any recent occurrences of this bug?

Comment 5 Alexander Chuzhoy 2017-07-25 10:56:42 EDT

We've implemented the w/a everywhere -  re-running the upload command.
In order to check it, will need to remove the w/a. Was there anything done to fix it?

Comment 6 Jiri Stransky 2017-07-26 07:18:58 EDT

There wasn't any fix, i still think the current information we have available suggests it's probably an internal infrastructure issue. When using the upstream registries, i never hit this nor did i come across anyone mention this.

Comment 7 Omri Hochman 2017-08-10 13:38:38 EDT

(In reply to Jiri Stransky from comment #4)
> Are there any recent occurrences of this bug?

Issue still occurs ..  need to re-run the command and it usually works the second time .

Comment 8 Omri Hochman 2017-08-10 13:55:04 EDT

(In reply to Omri Hochman from comment #7)
> (In reply to Jiri Stransky from comment #4)
> > Are there any recent occurrences of this bug?
> 
> Issue still occurs ..  need to re-run the command and it usually works the
> second time .
 
reproduced on Bare-Metal environment.

Comment 10 Jiri Stransky 2017-11-02 09:34:57 EDT

Martin and Dan added retry logic into the places that download containers:

https://github.com/openstack/tripleo-heat-templates/commit/d3b3361a76c2e8b188fa8e586d9fb7f3c60bb66f

https://github.com/openstack/tripleo-common/commit/2ff0842fcb27af43903d1738bff26bec558c966a

Specifically the latter might be relevant here. I still haven't hit the problem myself so i can't test if it fixes this very bug, but i think the patch is worth backporting either way. Proposed a stable/pike backport at https://review.openstack.org/517305

Comment 12 Alexander Chuzhoy 2017-11-15 12:45:24 EST

Verified:

Environment:
openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
instack-undercloud-7.4.3-2.el7ost.noarch


The reported issue doesn't reproduce.

Comment 16 errata-xmlrpc 2017-12-13 16:29:42 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.