Bug 1348031

Summary:	Can't push image to registry if disable the content-offload feature
Product:	OpenShift Container Platform	Reporter:	zhou ying <yinzhou>
Component:	Image Registry	Assignee:	Michal Minar <miminar>
Status:	CLOSED ERRATA	QA Contact:	Wei Sun <wsun>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.2.0	CC:	abhgupta, aos-bugs, tdawson, xtian
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: There's a bug in older registry version preventing it from working with a swift storage back-end while having content-offload feature turned off. Consequence: Registry was unusable with content-offload turned off and swift storage back-end. Fix: Updated registry version. New version has reworked storage drivers. Result: Registry is usable with configuration described above.	Story Points:	---
Clone Of:
Clones:	1382791 (view as bug list)		Environment:
Last Closed:	2016-09-27 09:38:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1382791

Description zhou ying 2016-06-20 02:04:38 UTC

Description of problem:
Config registry with Swift as storage backend, disable the content-offload feature, create a app and build image, the build will failed with error:
Failed to push image. Response from registry is: missing content-length header for request: https://172.30.138.252:5000/v2/zhouytt1/ruby-hello-world/blobs/sha256:10eeced9005a0de80b1077bb5be475ebc3227be76187f39bfb0c6c82474643bb

Version-Release number of selected component (if applicable):
openshift v3.2.1.2-1-g64665fc
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5


How reproducible:
always

Steps to Reproduce:
1. Config registry with Swift as storage backend;
2. Disable the content-offload feature:
  `oc env dc/docker-registry REGISTRY_STORAGE_REDIRECT_DISABLE=true`
3. Login OpenShift and create app:
  `oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git`
4. Check the build

Actual results:
4. Build failed with error:
builder.go:204] Error: build error: Failed to push image. Response from registry is: missing content-length header for request: https://172.30.138.252:5000/v2/zhouytt1/ruby-hello-world/blobs/sha256:10eeced9005a0de80b1077bb5be475ebc3227be
76187f39bfb0c6c82474643bb

Expected results:
Build succeed.

Additional info:

Comment 1 Abhishek Gupta 2016-06-20 16:40:15 UTC

Moving this to AEP/OSE since this pertains to Swift/OpenStack.

Comment 2 Michal Minar 2016-06-29 09:02:16 UTC

Just happened to me with s3 backend and 2.2.1 registry. So it's not swift specific error.
There is a fix upstream: https://github.com/docker/distribution/pull/1303

With docker registry rebased 2.4.0. This should work. Could you please retest with recent registry version?

Comment 3 Wei Sun 2016-06-30 03:10:40 UTC

We will re-test this bug when there is new puddle with recent registry version.

Comment 7 zhou ying 2016-07-15 06:45:42 UTC

blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1356823.

Comment 8 zhou ying 2016-08-02 10:40:05 UTC

Confirmed with ose3.3, the issue has fixed:
openshift version
openshift v3.3.0.13
kubernetes v1.3.0+57fb9ac
etcd 2.3.0+git


oc env po docker-registry-5-ia7j0 --list
# pods docker-registry-5-ia7j0, container registry
REGISTRY_HTTP_ADDR=:5000
REGISTRY_HTTP_NET=tcp
REGISTRY_HTTP_SECRET=tgrxM5Kspugc11TVwcMTrvEHiU0XEsTiqH1MuLvhFcc=
REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA=false
REGISTRY_CONFIGURATION_PATH=/etc/registryconfig/config.yml
REGISTRY_STORAGE_REDIRECT_DISABLE=true
[root@openshift-133 master]# oc logs po/docker-registry-5-ia7j0
time="2016-08-02T06:35:40.952355371-04:00" level=info msg="version=v2.4.1+unknown"

Comment 9 zhou ying 2016-09-01 08:25:16 UTC

confirmed with fork_ami_openshift3_miminar_295, and can not reproduce this issue

Comment 10 Michal Minar 2016-09-01 08:30:05 UTC

@yinzhou thanks!

Comment 12 errata-xmlrpc 2016-09-27 09:38:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933