Bug 1322762
| Summary: | sha256 Conflict while pull images after upgrade | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Anping Li <anli> | ||||
| Component: | docker | Assignee: | Antonio Murdaca <amurdaca> | ||||
| Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 7.2 | CC: | agoldste, anli, aos-bugs, bleanhar, dwalsh, jhonce, jokerman, lsm5, lsu, mmccomas, sjenning | ||||
| Target Milestone: | rc | Keywords: | Extras | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-05-12 15:17:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Hit same issue on another environment. But this time the bad pod is epa64mysqls2i, and the cakephp-mysql-example pod work well.
So prompt the severity to high. which will make some pod failed to started.
14m 48s 7 {kubelet osecontain-master1.example.com} spec.containers{eap-app} Normal Pulling pulling image "172.30.21.73:5000/epa64mysqls2i/eap-app@sha256:a51088b4e25343882520aef6f983fb4cdd063546b5a11331789e530cf42a148c"
11m 47s 6 {kubelet osecontain-master1.example.com} spec.containers{eap-app} Warning Failed Failed to pull image "172.30.21.73:5000/epa64mysqls2i/eap-app@sha256:a51088b4e25343882520aef6f983fb4cdd063546b5a11331789e530cf42a148c": Conflict: Digest sha256:a51088b4e25343882520aef6f983fb4cdd063546b5a11331789e530cf42a148c is already set to image a65a5a9f8576e7e281d1e92ef8161376614160886af5659617216a19ac4f18f0
The error is coming from Docker. It means that the node previously pulled 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 and recorded that it was associated with Docker v1 image ID a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c. On the pull that's not working, it's receiving the image manifest and for some reason it thinks that the same image manifest should be associated with a different v1 image ID. I don't have any theories yet as to why this is happening. Hit this issue again at the third Env. cakephpmysql is bad this time while epa64mysqls2i works well.
1h 4m 18 {kubelet minimaster.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.30.220.99:5000/cakephpmysql/cakephp-mysql-example@sha256:ad065f2bbb70fe15a5d48d594f7dcf270d4d94453a0ca7d639fa6bd467560b56": Conflict: Digest sha256:ad065f2bbb70fe15a5d48d594f7dcf270d4d94453a0ca7d639fa6bd467560b56 is already set to image f00d7d7174c35f8fe373c1074cd79320ca4647448782dd8c3c85471e1ee743ec
Something about https://github.com/projectatomic/docker/commit/504e67b867865a2835e8002c01087a2cfd7bfd0e is resulting in this error Andy, who should take this bug? I'm looking at it right now. A commit here that is in 1.9.1 that is not in 1.8.3 that refactors (possibly regresses) the digest tag code. Maybe the new code inverts the layerIDs? https://github.com/projectatomic/docker/commit/9b2426c606b92fa37b7484d9d884edd43352b69c When you pull by digest using 1.8, an entry is created in /var/lib/docker/repositories-devicemapper for the image repo -> digest -> v1 ID. After the upgrade to 1.9, when you pull the image by digest again (which should be a no-op), you get this error. It looks like the "strong ID" logic that computes deterministic IDs for the layers is coming up with a different v1 ID and then this error. I'm leaning toward this being an issue in 1.9, but there is a chance that the upgraded OSE registry coming from the upgrade from OSE 3.1 to 3.2 plays some part. This can be reproduced fairly easy with just docker. install docker 1.8.2 (yum downgrade docker, if needed) start local registry:2 w/ persistent storage docker pull busybox docker push localhost:5000/busybox (prints sha256 digest after push) docker pull localhost:5000/busybox@<digest> yum update docker (to 1.9.1) systemctl restart docker docker start registry docker pull localhost:5000/busybox@<digest> Trying to pull repository localhost:5000/busybox ... sha256:<digest>: Pulling from busybox failed Conflict: Digest sha256:<digest> is already set to image <v1 weak ID> See attachment for a my particular session. Created attachment 1143899 [details]
reproduce-procedure
just curious while I setup an env to reproduce this bz, is this happening with the upstream 1.9.1 docker as well? Asking because we did some edits in the past to the affected code. We didn't test upstream - only our RPMs alright, I see this in upstream 1.9.1 as well - will need to test more and see what's the cause. Found the root cause of this in 1.9.1 - basically the new 1.9.1 code compute the id (not the strong one) based on the blobSum(s) hashes (among other things) that are different between the ones we get from the localhost registry and the one we originally got from docker.io (and later tagged) so we get the wrong (weak) ID (this doesn't happen using docker.io because the busybox you pull it's *derived* by that but the busybox we use in our registry comes from the docker.io manifest). In 1.8.3 this check isn't performed of course. I'm not sure how to fix this, still trying to play around with the code. What's really interesting is that using upstream 1.8.3 -> projectatomic 1.9.1 doesn't trigger this issue - I was able to repull the image from my registry with the digest. So, what is really broken is probably our projectatomic 1.8.2 because it appears that projectatomic 1.9.1 is working fine when upgrading from upstream 1.8.3 (but again, it's broken after upgrading from projectatomic 1.8.2) We probably backported too much into projectatomic 1.8.2 or missed something from upstream 1.8.3. I'm still playing around with this though. I'll try playing with our patches in our 1.8.2 (and upstream 1.8.2 also). Just retested (note that we don't have a projectatomic docker 1.8.3): upstream 1.8.2 -> projectatomic 1.9.1 -> broken upstream 1.8.2 -> upstream 1.9.1 -> broken upstream 1.8.3 -> projectatomic 1.9.1 -> works upstream 1.8.3 -> upstream 1.9.1 -> works projectatomic 1.8.2 -> projectatomic 1.9.1 -> broken projectatomic 1.8.2 -> upstream 1.9.1 -> broken So I think we're clearly missing some patches from our projectatomic 1.8.2 - specifically from upstream 1.8.3. I'll look around to see if it's feasible to patch (again) our docker 1.9.1 to handle this correctly but I don't think so since it would require removing patches (or adjusting code in a way I don't feel comfortable at all). To me, it looks like https://github.com/docker/docker/pull/16953 is missing from our 1.8.2 backports (which belongs to upstream 1.8.3 though) and which seems to be the discriminating factor here (still have to test this out, but it requires a backport to 1.8.2 which we don't provide rpm anymore) Given we can't patch 1.9.1 to remove this check the only workaround I see here is to "docker rmi image@digest" the affected image and repull it again. So this is Release note issue then? Yes, we'll have to make this part of the OSE release notes. If there was a know what to detect affected images we could remove them during the upgrade automatically. From docker-1.8.2-3.el7.x86_64.rpm to docker-1.9.1-38.el7.x86_64.rpm In docker-1.8.2-3.el7.x86_64.rpm # docker pull busybox Using default tag: latest Trying to pull repository registry.access.redhat.com/busybox ... not found Trying to pull repository docker.io/library/busybox ... latest: Pulling from library/busybox 56ed16bd6310: Pull complete bc744c4ab376: Pull complete Digest: sha256:4a887a2326ec9e0fa90cce7b4764b0e627b5d6afcb81a3f73c85dc29cea00048 Status: Downloaded newer image for docker.io/busybox:latest # docker tag busybox myregistrydomain.com:5000/shatest # docker push myregistrydomain.com:5000/shatest The push refers to a repository [myregistrydomain.com:5000/shatest] (len: 1) bc744c4ab376: Pushed 56ed16bd6310: Pushed latest: digest: sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 size: 3202 # docker pull myregistrydomain.com:5000/shatest Using default tag: latest Trying to pull repository myregistrydomain.com:5000/shatest ... latest: Pulling from shatest 56ed16bd6310: Already exists bc744c4ab376: Already exists Digest: sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 Status: Image is up to date for myregistrydomain.com:5000/shatest:latest Upgrade to 1.9 #rpm -Uvh *.rpm # docker pull myregistrydomain.com:5000/shatest Using default tag: latest Trying to pull repository myregistrydomain.com:5000/shatest ... latest: Pulling from shatest 3cbb862e15c2: Pull complete 5d5461996ca8: Pull complete Digest: sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 Status: Downloaded newer image for myregistrydomain.com:5000/shatest:latest Move to verified (In reply to Luwen Su from comment #25) > From docker-1.8.2-3.el7.x86_64.rpm to docker-1.9.1-38.el7.x86_64.rpm > > In docker-1.8.2-3.el7.x86_64.rpm > # docker pull busybox > Using default tag: latest > Trying to pull repository registry.access.redhat.com/busybox ... not found > Trying to pull repository docker.io/library/busybox ... latest: Pulling from > library/busybox > 56ed16bd6310: Pull complete > bc744c4ab376: Pull complete > Digest: > sha256:4a887a2326ec9e0fa90cce7b4764b0e627b5d6afcb81a3f73c85dc29cea00048 > Status: Downloaded newer image for docker.io/busybox:latest > > # docker tag busybox myregistrydomain.com:5000/shatest > # docker push myregistrydomain.com:5000/shatest > The push refers to a repository [myregistrydomain.com:5000/shatest] (len: 1) > bc744c4ab376: Pushed > 56ed16bd6310: Pushed > latest: digest: > sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 > size: 3202 > # docker pull myregistrydomain.com:5000/shatest > Using default tag: latest > Trying to pull repository myregistrydomain.com:5000/shatest ... latest: > Pulling from shatest > 56ed16bd6310: Already exists > bc744c4ab376: Already exists > Digest: > sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 > Status: Image is up to date for myregistrydomain.com:5000/shatest:latest > > Upgrade to 1.9 > #rpm -Uvh *.rpm > # docker pull myregistrydomain.com:5000/shatest > Using default tag: latest > Trying to pull repository myregistrydomain.com:5000/shatest ... latest: > Pulling from shatest > 3cbb862e15c2: Pull complete > 5d5461996ca8: Pull complete > Digest: > sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 > Status: Downloaded newer image for myregistrydomain.com:5000/shatest:latest > > > Move to verified Sorry, the exact step to repro and verify are https://bugzilla.redhat.com/attachment.cgi?id=1143899 (specifically you aren't pulling by digest in your test) We still have a pending PR to fix this BZ https://github.com/projectatomic/docker/pull/141 We didn't merge the pull request, did we? I don't know. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-1034.html |
Description of problem: The pod cakephp-mysql-example report sha256 Conflict error after upgrade. only cakephp-mysql-example hit this issue, the other pod works well, for example cakephp-example. Pull this images by docker directly get same error. after delete the Conflict images , it can be pulled. No sure if it is for upgrade. please feel free to change the component type. Version-Release number of selected component (if applicable): openshift3/ose-deployer:v3.2.0.8 atomic-openshift-utils-3.0.68 How reproducible: Always in my environment. Steps to Reproduce: 1. setup containerlized env and customized the network_cidr osm_cluster_network_cidr="10.2.0.0/16" openshift_master_portal_net="172.31.0.0/16" 2. create cakephp-mysql-example applications 3. upgrade to ose3.2 4. check the pod status 5. scale up pod and check the pod status Actual results: 4.1) #oc get pods NAME READY STATUS RESTARTS AGE cakephp-mysql-example-1-prehook 0/1 Error 0 3h cakephp-mysql-example-2-gsuwd 0/1 ImagePullBackOff 0 33m mysql-1-hag2n 1/1 Running 0 33m 4.2) #oc describe pod cakephp-mysql-exsummary: sha256 Conflict when pull images after upgrade upgrade M M Description: The pod cakephp-mysql-example report sha256 Conflict error after upgrade. only cakephp-mysql-example hit this issue, the other pod works well, for example cakephp-example. Steps: 1. setup containerlized env and customized the network_cidr osm_cluster_network_cidr="10.2.0.0/16" openshift_master_portal_net="172.31.0.0/16" 2. create cakephp-mysql-example applications 3. upgrade to ose3.2 4. check the pod status 5. scale up pod and check the pod status Actual Result: 4.1) #oc get pods NAME READY STATUS RESTARTS AGE cakephp-mysql-example-1-prehook 0/1 Error 0 3h cakephp-mysql-example-2-gsuwd 0/1 ImagePullBackOff 0 33m mysql-1-hag2n 1/1 Running 0 33m 4.2) #oc describe pod cakephp-mysql-example-2-gsuwd Name: cakephp-mysql-example-2-gsuwd Namespace: cakephpmysql Node: osecontain-master1.example.com/10.66.78.90 Start Time: Thu, 31 Mar 2016 15:37:58 +0800 Labels: app=cakephp-mysql-example,deployment=cakephp-mysql-example-2,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example Status: Pending IP: 10.2.0.4 Controllers: ReplicationController/cakephp-mysql-example-2 Containers: cakephp-mysql-example: Container ID: Image: 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d Image ID: Port: 8080/TCP QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment Variables: DATABASE_SERVICE_NAME: mysql DATABASE_ENGINE: mysql DATABASE_NAME: default DATABASE_USER: cakephp DATABASE_PASSWORD: 5jR1QM2co1LxGp7A CAKEPHP_SECRET_TOKEN: hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA CAKEPHP_SECURITY_SALT: DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC CAKEPHP_SECURITY_CIPHER_SEED: 083731226737423233151050783143 OPCACHE_REVALIDATE_FREQ: 2 Conditions: Type Status Ready False Volumes: default-token-zkwja: Type: Secret (a volume populated by a Secret) SecretName: default-token-zkwja Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 35m 35m 6 {scheduler } FailedScheduling Failed for reason Region and possibly others 34m 34m 1 {scheduler } Scheduled Successfully assigned cakephp-mysql-example-2-gsuwd to osecontain-master1.example.com 34m 34m 1 {kubelet osecontain-master1.example.com} implicitly required container POD Pulled Container image "openshift3/ose-pod:v3.1.1.6" already present on machine 34m 34m 1 {kubelet osecontain-master1.example.com} implicitly required container POD Created Created with docker id 678a813fee80 34m 34m 1 {kubelet osecontain-master1.example.com} implicitly required container POD Started Started with docker id 678a813fee80 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Pulling pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Pulled Successfully pulled image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Created Created with docker id 08efb8d3d396 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Started Started with docker id 08efb8d3d396 8m 8m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request. details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials) 8m 8m 1 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request. details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials)" 10m 2m 6 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal Pulling pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 8m 2m 5 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780 8m 2m 5 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780" 8m 3s 34 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal BackOff Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 8m 3s 34 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d\"" 5.1) # oc get pods NAME READY STATUS RESTARTS AGE cakephp-mysql-example-1-05h21 0/1 ErrImagePull 0 1m cakephp-mysql-example-1-dhsfp 0/1 ImagePullBackOff 0 1m cakephp-mysql-example-1-prehook 0/1 Error 0 3h cakephp-mysql-example-2-gsuwd 0/1 ImagePullBackOff 0 35m mysql-1-hag2n 1/1 Running 0 35 5.2) #oc describe pod cakephp-mysql-example-1-dhsfp Name: cakephp-mysql-example-1-dhsfp Namespace: cakephpmysql Node: osecontain-master1.example.com/10.66.78.90 Start Time: Thu, 31 Mar 2016 16:11:06 +0800 Labels: app=cakephp-mysql-example,deployment=cakephp-mysql-example-1,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example Status: Pending IP: 10.2.0.6 Controllers: ReplicationController/cakephp-mysql-example-1 Containers: cakephp-mysql-example: Container ID: Image: 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 Image ID: Port: 8080/TCP QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment Variables: DATABASE_SERVICE_NAME: mysql DATABASE_ENGINE: mysql DATABASE_NAME: default DATABASE_USER: cakephp DATABASE_PASSWORD: 5jR1QM2co1LxGp7A CAKEPHP_SECRET_TOKEN: hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA CAKEPHP_SECURITY_SALT: DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC CAKEPHP_SECURITY_CIPHER_SEED: 083731226737423233151050783143 OPCACHE_REVALIDATE_FREQ: 2 Conditions: Type Status Ready False Volumes: default-token-zkwja: Type: Secret (a volume populated by a Secret) SecretName: default-token-zkwja Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 36s 36s 1 {default-scheduler } Normal Scheduled Successfully assigned cakephp-mysql-example-1-dhsfp to osecontain-master1.example.com 24s 24s 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal BackOff Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16" 24s 24s 1 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16\"" 30s 9s 2 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal Pulling pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16" 25s 9s 2 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16": Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c 25s 9s 2 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c" ample-2-gsuwd Name: cakephp-mysql-example-2-gsuwd Namespace: cakephpmysql Node: osecontain-master1.example.com/10.66.78.90 Start Time: Thu, 31 Mar 2016 15:37:58 +0800 Labels: app=cakephp-mysql-example,deployment=cakephp-mysql-example-2,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example Status: Pending IP: 10.2.0.4 Controllers: ReplicationController/cakephp-mysql-example-2 Containers: cakephp-mysql-example: Container ID: Image: 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d Image ID: Port: 8080/TCP QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment Variables: DATABASE_SERVICE_NAME: mysql DATABASE_ENGINE: mysql DATABASE_NAME: default DATABASE_USER: cakephp DATABASE_PASSWORD: 5jR1QM2co1LxGp7A CAKEPHP_SECRET_TOKEN: hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA CAKEPHP_SECURITY_SALT: DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC CAKEPHP_SECURITY_CIPHER_SEED: 083731226737423233151050783143 OPCACHE_REVALIDATE_FREQ: 2 Conditions: Type Status Ready False Volumes: default-token-zkwja: Type: Secret (a volume populated by a Secret) SecretName: default-token-zkwja Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 35m 35m 6 {scheduler } FailedScheduling Failed for reason Region and possibly others 34m 34m 1 {scheduler } Scheduled Successfully assigned cakephp-mysql-example-2-gsuwd to osecontain-master1.example.com 34m 34m 1 {kubelet osecontain-master1.example.com} implicitly required container POD Pulled Container image "openshift3/ose-pod:v3.1.1.6" already present on machine 34m 34m 1 {kubelet osecontain-master1.example.com} implicitly required container POD Created Created with docker id 678a813fee80 34m 34m 1 {kubelet osecontain-master1.example.com} implicitly required container POD Started Started with docker id 678a813fee80 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Pulling pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Pulled Successfully pulled image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Created Created with docker id 08efb8d3d396 34m 34m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Started Started with docker id 08efb8d3d396 8m 8m 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request. details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials) 8m 8m 1 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request. details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials)" 10m 2m 6 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal Pulling pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 8m 2m 5 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780 8m 2m 5 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780" 8m 3s 34 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal BackOff Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d" 8m 3s 34 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d\"" 5.1) # oc get pods NAME READY STATUS RESTARTS AGE cakephp-mysql-example-1-05h21 0/1 ErrImagePull 0 1m cakephp-mysql-example-1-dhsfp 0/1 ImagePullBackOff 0 1m cakephp-mysql-example-1-prehook 0/1 Error 0 3h cakephp-mysql-example-2-gsuwd 0/1 ImagePullBackOff 0 35m mysql-1-hag2n 1/1 Running 0 35 5.2) #oc describe pod cakephp-mysql-example-1-dhsfp Name: cakephp-mysql-example-1-dhsfp Namespace: cakephpmysql Node: osecontain-master1.example.com/10.66.78.90 Start Time: Thu, 31 Mar 2016 16:11:06 +0800 Labels: app=cakephp-mysql-example,deployment=cakephp-mysql-example-1,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example Status: Pending IP: 10.2.0.6 Controllers: ReplicationController/cakephp-mysql-example-1 Containers: cakephp-mysql-example: Container ID: Image: 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 Image ID: Port: 8080/TCP QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment Variables: DATABASE_SERVICE_NAME: mysql DATABASE_ENGINE: mysql DATABASE_NAME: default DATABASE_USER: cakephp DATABASE_PASSWORD: 5jR1QM2co1LxGp7A CAKEPHP_SECRET_TOKEN: hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA CAKEPHP_SECURITY_SALT: DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC CAKEPHP_SECURITY_CIPHER_SEED: 083731226737423233151050783143 OPCACHE_REVALIDATE_FREQ: 2 Conditions: Type Status Ready False Volumes: default-token-zkwja: Type: Secret (a volume populated by a Secret) SecretName: default-token-zkwja Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 36s 36s 1 {default-scheduler } Normal Scheduled Successfully assigned cakephp-mysql-example-1-dhsfp to osecontain-master1.example.com 24s 24s 1 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal BackOff Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16" 24s 24s 1 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16\"" 30s 9s 2 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Normal Pulling pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16" 25s 9s 2 {kubelet osecontain-master1.example.com} spec.containers{cakephp-mysql-example} Warning Failed Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16": Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c 25s 9s 2 {kubelet osecontain-master1.example.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c" Expected results: Additional info: