Bug 1408661
| Summary: | [3.4] Fail to push built image to registry due to "manifest blob unknown: blob unknown to registry" without define output image via oc new-build | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wenjing Zheng <wzheng> | |
| Component: | Image Registry | Assignee: | Alexey Gladkov <agladkov> | |
| Status: | CLOSED ERRATA | QA Contact: | Wei Sun <wsun> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 3.4.0 | CC: | aos-bugs, bparees, cewong, haowang, mfojtik, tdawson, wzheng, xtian | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1411161 1411162 (view as bug list) | Environment: | ||
| Last Closed: | 2017-01-18 12:57:33 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1411161, 1411162 | |||
Same as issue https://github.com/openshift/origin/issues/12329 Changing the severity to high. This is reproducible every time we create a build that is pushing to the same image stream (not same tag) AND the base layer is not already in the registry. Just a little bit more detail... here is the log of the failed Docker build: Step 1 : FROM centos@sha256:c577af3197aacedf79c5a204cd7f493c8e07ffbce7f88f7600bf19c688c38799 ---> 67591570dd29 Step 2 : RUN yum install -y httpd ---> Using cache ---> 74238d506938 Step 3 : ENV "OPENSHIFT_BUILD_NAME" "centos-1" "OPENSHIFT_BUILD_NAMESPACE" "myproject" ---> Using cache ---> a4c975f52f9f Successfully built a4c975f52f9f Pushing image 172.30.13.125:5000/myproject/centos:latest ... Pushed 1/2 layers, 53% complete Pushed 2/2 layers, 100% complete Registry server Address: Registry server User Name: serviceaccount Registry server Email: serviceaccount Registry server Password: <<non-empty>> error: build error: Failed to push image: errors: manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry The FROM of the Docker build is pointing to imagestream centos, tag "7". It is trying to push to the same imagestream, but tag "latest". > The FROM of the Docker build is pointing to imagestream centos, tag "7". It
> is trying to push to the same imagestream, but tag "latest".
Hi, Cesar, after the create the build, the imagestream don't have the latet tag created, does that matters?
from the bug description:
$ oc get is
NAME DOCKER REPO TAGS UPDATED
centos 172.30.69.164:5000/wzheng4/centos 7 5 minutes ago
But if define the to image to tag 7, build still fail due to cannot push built image: Pushing image 172.30.74.209:5000/wzheng/centos:7 ... Pushed 1/2 layers, 52% complete Pushed 2/2 layers, 100% complete Registry server Address: Registry server User Name: serviceaccount Registry server Email: serviceaccount Registry server Password: <<non-empty>> error: build error: Failed to push image: errors: manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry $ oc describe bc myapp Name: myapp Namespace: wzheng Created: 13 minutes ago Labels: build=myapp Annotations: openshift.io/generated-by=OpenShiftNewBuild Latest Version: 1 Strategy: Docker Dockerfile: FROM centos:7 RUN yum install -y httpd From Image: ImageStreamTag centos:7 Output to: ImageStreamTag centos:7 Empty Source: no input source provided Build Run Policy: Serial Triggered by: Config, ImageChange Webhook GitHub: URL: https://10.8.175.105:8443/oapi/v1/namespaces/wzheng/buildconfigs/myapp/webhooks/P5vvf42ZduR2zysCnekl/github Webhook Generic: URL: https://10.8.175.105:8443/oapi/v1/namespaces/wzheng/buildconfigs/myapp/webhooks/KQlYx-_PIAz-XsKSGjvc/generic AllowEnv: false Build Status Duration Creation Time myapp-1 failed 31s 2017-01-04 10:48:39 +0800 CST No events. (In reply to Wang Haoran from comment #4) > > The FROM of the Docker build is pointing to imagestream centos, tag "7". It > > is trying to push to the same imagestream, but tag "latest". > > Hi, Cesar, after the create the build, the imagestream don't have the latet > tag created, does that matters? The tag is only created by the push when the build succeeds. Because it is failing, the tag doesn't get created. I wanted to also note that if the 'from' imagestream is different than the 'to' imagestream, then the initial push does succeed: oc new-build -D $'FROM centos:7\nRUN yum install -y httpd' --name=mycentos Therefore the problem happens only when using the same imagestream for both 'from' and 'to' it also only seems to happen on the first push of the image which makes me suspect it has something to do with layer reuse and perhaps the cross-project layer sharing (though there is nothing cross project about this particular scenario). (that is, if you successfully run the build/push using a different input/output imagestreams, you can then successfully run that same build using matched input/output imagestreams). Verified on below version: openshift v3.4.0.39 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066 |
Description of problem: Build fails to push to registry when from image and to image are the same for below error: Registry server Email: serviceaccount Registry server Password: <<non-empty>> error: build error: Failed to push image: errors: manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry manifest blob unknown: blob unknown to registry When describe the buildconfig, below error appears: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 40m 40m 11 {build-config-controller } Warning BuildConfigInstantiateFailed error instantiating Build from BuildConfig wzheng1/centos: Error resolving ImageStreamTag centos:7 in namespace wzheng1: imagestreamtags "centos:7" not found $ oc get is NAME DOCKER REPO TAGS UPDATED centos 172.30.69.164:5000/wzheng4/centos 7 5 minutes ago Version-Release number of selected component (if applicable): openshift v3.4.0.37+3b76456-1 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 How reproducible: always Steps to Reproduce: 1. oc new-build -D $'FROM centos:7\nRUN yum install -y httpd' 2. 3. Actual results: Build failed Expected results: Build should succeed Additional info: