Bug 1348031 – Can't push image to registry if disable the content-offload feature

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1348031 - Can't push image to registry if disable the content-offload feature

Summary:	Can't push image to registry if disable the content-offload feature

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Image Registry (Show other bugs)
Sub Component:
Version:	3.2.0
Hardware:	Unspecified Unspecified

Priority	medium Severity medium
Target Milestone:	---
Target Release:	---
Assigned To:	Michal Minar
QA Contact:	Wei Sun
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:	1382791
	Show dependency tree / graph

Reported:	2016-06-19 22:04 EDT by zhou ying
Modified:	2017-03-08 13 EST (History)
CC List:	4 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: There's a bug in older registry version preventing it from working with a swift storage back-end while having content-offload feature turned off. Consequence: Registry was unusable with content-offload turned off and swift storage back-end. Fix: Updated registry version. New version has reworked storage drivers. Result: Registry is usable with configuration described above.
Story Points:	---
Clone Of:
Clones:	1382791 (view as bug list)
Environment:
Last Closed:	2016-09-27 05:38:30 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:1933	normal	SHIPPED_LIVE	Red Hat OpenShift Container Platform 3.3 Release Advisory	2016-09-27 09:24:36 EDT

Groups: None (edit)

Description zhou ying 2016-06-19 22:04:38 EDT

Description of problem:
Config registry with Swift as storage backend, disable the content-offload feature, create a app and build image, the build will failed with error:
Failed to push image. Response from registry is: missing content-length header for request: https://172.30.138.252:5000/v2/zhouytt1/ruby-hello-world/blobs/sha256:10eeced9005a0de80b1077bb5be475ebc3227be76187f39bfb0c6c82474643bb

Version-Release number of selected component (if applicable):
openshift v3.2.1.2-1-g64665fc
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5


How reproducible:
always

Steps to Reproduce:
1. Config registry with Swift as storage backend;
2. Disable the content-offload feature:
  `oc env dc/docker-registry REGISTRY_STORAGE_REDIRECT_DISABLE=true`
3. Login OpenShift and create app:
  `oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git`
4. Check the build

Actual results:
4. Build failed with error:
builder.go:204] Error: build error: Failed to push image. Response from registry is: missing content-length header for request: https://172.30.138.252:5000/v2/zhouytt1/ruby-hello-world/blobs/sha256:10eeced9005a0de80b1077bb5be475ebc3227be
76187f39bfb0c6c82474643bb

Expected results:
Build succeed.

Additional info:

Comment 1 Abhishek Gupta 2016-06-20 12:40:15 EDT

Moving this to AEP/OSE since this pertains to Swift/OpenStack.

Comment 2 Michal Minar 2016-06-29 05:02:16 EDT

Just happened to me with s3 backend and 2.2.1 registry. So it's not swift specific error.
There is a fix upstream: https://github.com/docker/distribution/pull/1303

With docker registry rebased 2.4.0. This should work. Could you please retest with recent registry version?

Comment 3 Wei Sun 2016-06-29 23:10:40 EDT

We will re-test this bug when there is new puddle with recent registry version.

Comment 7 zhou ying 2016-07-15 02:45:42 EDT

blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1356823.

Comment 8 zhou ying 2016-08-02 06:40:05 EDT

Confirmed with ose3.3, the issue has fixed:
openshift version
openshift v3.3.0.13
kubernetes v1.3.0+57fb9ac
etcd 2.3.0+git


oc env po docker-registry-5-ia7j0 --list
# pods docker-registry-5-ia7j0, container registry
REGISTRY_HTTP_ADDR=:5000
REGISTRY_HTTP_NET=tcp
REGISTRY_HTTP_SECRET=tgrxM5Kspugc11TVwcMTrvEHiU0XEsTiqH1MuLvhFcc=
REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA=false
REGISTRY_CONFIGURATION_PATH=/etc/registryconfig/config.yml
REGISTRY_STORAGE_REDIRECT_DISABLE=true
[root@openshift-133 master]# oc logs po/docker-registry-5-ia7j0
time="2016-08-02T06:35:40.952355371-04:00" level=info msg="version=v2.4.1+unknown"

Comment 9 zhou ying 2016-09-01 04:25:16 EDT

confirmed with fork_ami_openshift3_miminar_295, and can not reproduce this issue

Comment 10 Michal Minar 2016-09-01 04:30:05 EDT

@yinzhou thanks!

Comment 12 errata-xmlrpc 2016-09-27 05:38:30 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933

Note You need to log in before you can comment on or make changes to this bug.