Cause:
We pass all requests (including HEAD) to the remote service if the pullthrough is enabled. On the other hand when docker client pushes the manifest we check the presence of all these layers locally. The client checks a blob existance by HEAD request before sending it to the server.
If client image is based on the imported image (but not present in local registry) dockerregistry will say that it has all the layers from the base image. In this case docker client never send them to server, but manifest verification requires them locally. It means that the verification will always fail for remote layers.
Fix:
Manifest verification must to take into account the possibility that the layers may not be local and check them on remote registry server before before give up.
We can't use pullthroughBlobStore because verification happens in ManifestService. So we need to move common code that gets the blobs from the remote server to BlobGetterService and use it for pullthrough and for verification.
Can reproduce with 3.2 as below: [root@openshift-130 ~]# oc logs builds/centos-1 I0203 06:06:47.924209 1 builder.go:57] Master version "v3.2.1.23", Builder version "v3.2.1.23" I0203 06:06:47.947773 1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:92233720368547, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:92233720368547} Step 1 : FROM centos@sha256:71574605afb91eb5adb332fd73e4baff2fc7b1bed8364cfe018b28ccdf0e76f3 Trying to pull repository registry.access.redhat.com/centos ... Trying to pull repository docker.io/library/centos ... F0203 06:06:50.434471 1 builder.go:204] Error: build error: manifest unknown: manifest unknown [root@openshift-130 ~]# openshift version openshift v3.2.1.23 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5