Bug 1396484

Summary: Manifest unknown error when pulling images in 3.3.1 environment
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: Image RegistryAssignee: Alexey Gladkov <agladkov>
Status: CLOSED NOTABUG QA Contact: Wei Sun <wsun>
Severity: high Docs Contact:
Priority: high    
Version: 3.3.1CC: aos-bugs, erich, jkaur, maszulik, mfojtik, pweil
Target Milestone: ---Keywords: Unconfirmed
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-02 05:06:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jaspreet Kaur 2016-11-18 13:12:14 UTC 
Description of problem: Build fails with manifest unknown error after upgrading to Openshift 3.3.1.3 and docker 1.10.3. The old images are already in the docker registry and were build with docker 1.9. Setting below parameter is not working :

REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2

https://docs.openshift.com/container-platform/3.3/install_config/registry/extended_registry_configuration.html#docker-registry-configuration-reference-middleware

The only solution is : rebuilding the images but it leaves us in bad position, since previously built images can't be reused.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: Build fails with ImagePullBackOff

 1m		25s		3	{kubelet osen02-ppe.brreg.no}	spec.containers{config-cloudserver}	Warning		Failed		Failed to pull image "172.30.49.59:5000/rofs-ppe/config-cloudserver@sha256:2995c37c3512b523d69fe1189b8b9865c4ec94ffdc92209170c37d6f61b0f8be": manifest unknown: manifest unknown




Expected results:  There should be some migration procedure to avoid the issue.


Additional info:
Comment 1 Alexey Gladkov 2016-11-18 14:57:50 UTC 
I can't reproduce it. After command

oc env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true

the error:

time="2016-11-18T14:49:03.160239911Z" level=error msg="response completed with error" err.code="manifest invalid" err.detail="manifest V2 schema 2 not allowed" err.message="manifest invalid" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="172.30.121.25:5000" http.request.id=7394f036-0322-4f68-861a-0364b12b4372 http.request.method=PUT http.request.remoteaddr="10.34.129.55:47986" http.request.uri="/v2/tmp/busybox/manifests/latest" http.request.useragent="docker/1.10.3 go/go1.6.3 kernel/4.8.6-201.fc24.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.209911ms http.response.status=400 http.response.written=82 instance.id=25e4ada3-e973-4186-ae7c-51c8e701a7ea vars.name="tmp/busybox" vars.reference=latest

goes away and the first attempt to push the manifest (schema v2) ends with success.

Can you give more information how to reproduce it ?