Bug 1411162 - [3.2] Fail to push built image to registry due to "manifest blob unknown: blob unknown to registry" without define output image via oc new-build
Summary: [3.2] Fail to push built image to registry due to "manifest blob unknown: blo...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.2.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.2.1
Assignee: Alexey Gladkov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On: 1408661
Blocks: 1411161
TreeView+ depends on / blocked
 
Reported: 2017-01-08 22:09 UTC by Scott Dodson
Modified: 2017-03-02 14:49 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: We pass all requests (including HEAD) to the remote service if the pullthrough is enabled. On the other hand when docker client pushes the manifest we check the presence of all these layers locally. The client checks a blob existance by HEAD request before sending it to the server. If client image is based on the imported image (but not present in local registry) dockerregistry will say that it has all the layers from the base image. In this case docker client never send them to server, but manifest verification requires them locally. It means that the verification will always fail for remote layers. Fix: Manifest verification must to take into account the possibility that the layers may not be local and check them on remote registry server before before give up. We can't use pullthroughBlobStore because verification happens in ManifestService. So we need to move common code that gets the blobs from the remote server to BlobGetterService and use it for pullthrough and for verification.
Clone Of: 1408661
Environment:
Last Closed: 2017-03-02 14:49:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Comment 3 Wenjing Zheng 2017-02-03 06:26:36 UTC 
Can reproduce with 3.2 as below:
[root@openshift-130 ~]# oc logs builds/centos-1
I0203 06:06:47.924209       1 builder.go:57] Master version "v3.2.1.23", Builder version "v3.2.1.23"
I0203 06:06:47.947773       1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:92233720368547, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:92233720368547}
Step 1 : FROM centos@sha256:71574605afb91eb5adb332fd73e4baff2fc7b1bed8364cfe018b28ccdf0e76f3
Trying to pull repository registry.access.redhat.com/centos ... 
Trying to pull repository docker.io/library/centos ... 
F0203 06:06:50.434471       1 builder.go:204] Error: build error: manifest unknown: manifest unknown
[root@openshift-130 ~]# openshift version
openshift v3.2.1.23
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5
Comment 4 Alexey Gladkov 2017-02-08 12:58:09 UTC 
what docker version are you using ?
Comment 5 Wenjing Zheng 2017-02-09 02:11:45 UTC 
Here is my docker version:
# docker version
Client:
 Version:         1.12.5
 API version:     1.24
 Package version: docker-common-1.12.5-14.el7.x86_64
 Go version:      go1.7.4
 Git commit:      047e51b/1.12.5
 Built:           Wed Jan 11 17:53:20 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.5
 API version:     1.24
 Package version: docker-common-1.12.5-14.el7.x86_64
 Go version:      go1.7.4
 Git commit:      047e51b/1.12.5
 Built:           Wed Jan 11 17:53:20 2017
 OS/Arch:         linux/amd64
Note You need to log in before you can comment on or make changes to this bug.