Bug 1348031 - Can't push image to registry if disable the content-offload feature
Summary: Can't push image to registry if disable the content-offload feature
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Michal Minar
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks: 1382791
TreeView+ depends on / blocked
 
Reported: 2016-06-20 02:04 UTC by zhou ying
Modified: 2017-03-08 18:26 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: There's a bug in older registry version preventing it from working with a swift storage back-end while having content-offload feature turned off. Consequence: Registry was unusable with content-offload turned off and swift storage back-end. Fix: Updated registry version. New version has reworked storage drivers. Result: Registry is usable with configuration described above.
Clone Of:
: 1382791 (view as bug list)
Environment:
Last Closed: 2016-09-27 09:38:30 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Description zhou ying 2016-06-20 02:04:38 UTC 
Description of problem:
Config registry with Swift as storage backend, disable the content-offload feature, create a app and build image, the build will failed with error:
Failed to push image. Response from registry is: missing content-length header for request: https://172.30.138.252:5000/v2/zhouytt1/ruby-hello-world/blobs/sha256:10eeced9005a0de80b1077bb5be475ebc3227be76187f39bfb0c6c82474643bb

Version-Release number of selected component (if applicable):
openshift v3.2.1.2-1-g64665fc
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5


How reproducible:
always

Steps to Reproduce:
1. Config registry with Swift as storage backend;
2. Disable the content-offload feature:
  `oc env dc/docker-registry REGISTRY_STORAGE_REDIRECT_DISABLE=true`
3. Login OpenShift and create app:
  `oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git`
4. Check the build

Actual results:
4. Build failed with error:
builder.go:204] Error: build error: Failed to push image. Response from registry is: missing content-length header for request: https://172.30.138.252:5000/v2/zhouytt1/ruby-hello-world/blobs/sha256:10eeced9005a0de80b1077bb5be475ebc3227be
76187f39bfb0c6c82474643bb

Expected results:
Build succeed.

Additional info:
Comment 1 Abhishek Gupta 2016-06-20 16:40:15 UTC 
Moving this to AEP/OSE since this pertains to Swift/OpenStack.
Comment 2 Michal Minar 2016-06-29 09:02:16 UTC 
Just happened to me with s3 backend and 2.2.1 registry. So it's not swift specific error.
There is a fix upstream: https://github.com/docker/distribution/pull/1303

With docker registry rebased 2.4.0. This should work. Could you please retest with recent registry version?
Comment 3 Wei Sun 2016-06-30 03:10:40 UTC 
We will re-test this bug when there is new puddle with recent registry version.
Comment 7 zhou ying 2016-07-15 06:45:42 UTC 
blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1356823.
Comment 8 zhou ying 2016-08-02 10:40:05 UTC 
Confirmed with ose3.3, the issue has fixed:
openshift version
openshift v3.3.0.13
kubernetes v1.3.0+57fb9ac
etcd 2.3.0+git


oc env po docker-registry-5-ia7j0 --list
# pods docker-registry-5-ia7j0, container registry
REGISTRY_HTTP_ADDR=:5000
REGISTRY_HTTP_NET=tcp
REGISTRY_HTTP_SECRET=tgrxM5Kspugc11TVwcMTrvEHiU0XEsTiqH1MuLvhFcc=
REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA=false
REGISTRY_CONFIGURATION_PATH=/etc/registryconfig/config.yml
REGISTRY_STORAGE_REDIRECT_DISABLE=true
[root@openshift-133 master]# oc logs po/docker-registry-5-ia7j0
time="2016-08-02T06:35:40.952355371-04:00" level=info msg="version=v2.4.1+unknown"
Comment 9 zhou ying 2016-09-01 08:25:16 UTC 
confirmed with fork_ami_openshift3_miminar_295, and can not reproduce this issue
Comment 10 Michal Minar 2016-09-01 08:30:05 UTC 
@yinzhou thanks!
Comment 12 errata-xmlrpc 2016-09-27 09:38:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933
Note You need to log in before you can comment on or make changes to this bug.