Bug 1277046 - ImagePullBackOff for recreate-example after tagging a new version of the example as the latest tag
ImagePullBackOff for recreate-example after tagging a new version of the exam...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Image Registry (Show other bugs)
3.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Maciej Szulik
Wei Sun
:
Depends On:
Blocks: 1289358
  Show dependency treegraph
 
Reported: 2015-11-02 02:54 EST by Yan Du
Modified: 2016-05-12 13:09 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1289358 (view as bug list)
Environment:
Last Closed: 2016-05-12 13:09:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yan Du 2015-11-02 02:54:18 EST
Description of problem:
ImagePullBackOff for recreate-example after tagging a new version of the example as the latest tag


Version-Release number of selected component (if applicable):
oc v3.0.2.905
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4


How reproducible
Always


Steps to Reproduce:

1. oc create -f examples/deployment/recreate-example.yaml

2. Browse to the application at recreate-example.<project>.<router_domain> to verify you see the 'v1' image

3. Trigger a new deployment automatically by tagging a new version of the example as the latest tag:

# oc tag recreate-example:v2 recreate-example:latest
4. oc get pod



Actual results:

step3:
$ oc tag recreate-example:v2 recreate-example:latest
Tag recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b set up to track tag duyan/recreate-example:latest.


step4:
$ oc get pod
NAME                       READY     STATUS             RESTARTS   AGE
recreate-example-2-ca8gd   0/1       ImagePullBackOff   0          4m
recreate-example-2-obczf   0/1       ImagePullBackOff   0          4m



Expected results:
The deployment could be completed successfully and pod running well



Additional info:

OpenShift:origin $ oc describe pod recreate-example-2-ca8gd
Name:                recreate-example-2-ca8gd
Namespace:            duyan
Image(s):            172.30.167.160:5000/duyan/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
Node:                openshift-146.lab.eng.nay.redhat.com/10.66.79.146
Start Time:            Mon, 02 Nov 2015 13:31:25 +0800
Labels:                deployment=recreate-example-2,deploymentconfig=recreate-example
Status:                Pending
Reason:                
Message:            
IP:                10.1.1.13
Replication Controllers:    recreate-example-2 (2/2 replicas created)
Containers:
  deployment-example:
    Container ID:    
    Image:        172.30.167.160:5000/duyan/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
    Image ID:        
    QoS Tier:
      cpu:        BestEffort
      memory:        BestEffort
    State:        Waiting
      Reason:        ImagePullBackOff
    Ready:        False
    Restart Count:    0
    Environment Variables:
Conditions:
  Type        Status
  Ready     False 
Volumes:
  default-token-tcm08:
    Type:    Secret (a secret that should populate this volume)
    SecretName:    default-token-tcm08
Events:
  FirstSeen    LastSeen    Count    From                        SubobjectPath                Reason        Message
  ─────────    ────────    ─────    ────                        ─────────────                ──────        ───────
  7m        7m        1    {scheduler }                                        Scheduled    Successfully assigned recreate-example-2-ca8gd to openshift-146.lab.eng.nay.redhat.com
  7m        7m        1    {kubelet openshift-146.lab.eng.nay.redhat.com}    implicitly required container POD    Pulled        Container image "registry.access.redhat.com/aep3_beta/aep-pod:v3.0.2.905" already present on machine
  7m        7m        1    {kubelet openshift-146.lab.eng.nay.redhat.com}    implicitly required container POD    Created        Created with docker id c81f996557e2
  7m        7m        1    {kubelet openshift-146.lab.eng.nay.redhat.com}    implicitly required container POD    Started        Started with docker id c81f996557e2
  7m        7m        1    {kubelet openshift-146.lab.eng.nay.redhat.com}    spec.containers{deployment-example}    Failed        Failed to pull image "172.30.167.160:5000/duyan/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b": manifest unknown: manifest unknown
  7m        7m        1    {kubelet openshift-146.lab.eng.nay.redhat.com}    spec.containers{deployment-example}    Back-off    Back-off pulling image "172.30.167.160:5000/duyan/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b"
  7m        7m        2    {kubelet openshift-146.lab.eng.nay.redhat.com}    spec.containers{deployment-example}    Pulling        pulling image "172.30.167.160:5000/duyan/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b"
Comment 1 Yan Du 2015-11-13 01:59:04 EST
Here is the latest result which test against latest origin/OSE env

Origin
oc v1.0.8-28-g0ab00df
kubernetes v1.1.0-origin-1107-g4c8e6f4

OSE
oc v3.1.0.4-5-gebe80f5
kubernetes v1.1.0-origin-1107-g4c8e6f4

# oc tag recreate-example:v2 recreate-example:latest
Tag recreate-example:latest set to recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b.

# oc get pod
NAME                       READY     STATUS             RESTARTS   AGE
recreate-example-2-52d9n   0/1       ImagePullBackOff   0          1m
recreate-example-2-l12vo   0/1       ImagePullBackOff   0          1m
Comment 2 Dan Mace 2015-11-13 09:00:01 EST
The deployment is using the newly tagged image, so this doesn't seem like a deployments issue. Reassigning to node since the Kubelet is responsible for pulling.
Comment 3 Andy Goldstein 2015-11-13 10:01:01 EST
If you run 'oc get images' (need to be cluster-admin), do you see one whose name is sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b?
Comment 4 Jan Chaloupka 2015-11-13 10:14:51 EST
What version of docker do you use? The one built in Fedora or the one built form upstream (github.com/docker/docker) sources?
Comment 5 Andy Goldstein 2015-11-13 13:53:13 EST
This appears to be a bug in the way that 'oc tag' works:

*** Step 1: run oc tag

vagrant@localhost:~/go/src/github.com/openshift/origin (master) oc tag recreate-example:v2 recreate-example:latest
Tag recreate-example:latest set to recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b.

*** Step 2: check the ImageStream (only showing status.tags here):

  status:
    dockerImageRepository: 172.30.138.254:5000/default/recreate-example
    tags:
    - items:
      - created: 2015-11-13T18:40:31Z
        dockerImageReference: 172.30.138.254:5000/default/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
        image: sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
      - created: 2015-11-13T18:39:27Z
        dockerImageReference: openshift/deployment-example@sha256:ea9135488f323060cb18ab3ec06286cd49e4b3a611fce1a6a442651ecf421f99
        image: sha256:ea9135488f323060cb18ab3ec06286cd49e4b3a611fce1a6a442651ecf421f99
      tag: latest
    - items:
      - created: 2015-11-13T18:39:26Z
        dockerImageReference: openshift/deployment-example@sha256:c505b916f7e5143a356ff961f2c21aee40fbd2cd906c1e3feeb8d5e978da284b
        image: sha256:c505b916f7e5143a356ff961f2c21aee40fbd2cd906c1e3feeb8d5e978da284b
      tag: v1
    - items:
      - created: 2015-11-13T18:39:26Z
        dockerImageReference: openshift/deployment-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
        image: sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
      tag: v2

*** Problem:

Note that the current dockerImageReference for "latest" should match the value from the "v2" tag (openshift/deployment-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b). Instead, it ended up setting it to 172.30.138.254:5000/default/recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b, which is <registry>/<project>/<image stream>@<sha256 of actual image>.
Comment 6 Andy Goldstein 2015-11-13 14:06:39 EST
So what's happening is 'oc tag' is setting this in spec.tags:

spec:
  dockerImageRepository: openshift/deployment-example
  tags:
  - from:
      kind: ImageStreamImage
      name: recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
    name: latest

Notice that it's an ImageStreamImage. Next, inside pkg/image/registry/imagestream/strategy.go, it converts the ImageStreamImage reference into a pull spec here: https://github.com/openshift/origin/blob/3372e655695c8ed82ba0fc1dfa2a1d4c4fc7e74e/pkg/image/registry/imagestream/strategy.go#L232-L245.

Unfortunately, this logic assumes that all ImageStreamImage references are to images inside the integrated registry. That is not the case here. Instead, we have an ImageStream that has imported its tag data from the Docker Hub. What really should be happening is the pull spec should be the value from status.tags[<tag>].dockerImageReference, where the value for .image matches the ImageStreamImage value.

For example, given this status tag:

 - items:
      - created: 2015-11-13T18:39:26Z
        dockerImageReference: openshift/deployment-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
        image: sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b
      tag: v2

The pull spec should be taken from "dockerImageReference: openshift/deployment-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b".
Comment 7 Andy Goldstein 2015-11-13 14:33:21 EST
This may help resolve it: https://gist.github.com/ncdc/736df88061bc7bc17470. Tests are definitely needed to ensure we don't break anything else w/this fix :-)
Comment 8 Maciej Szulik 2015-11-18 10:01:26 EST
The PR for this is here https://github.com/openshift/origin/pull/5931
Comment 9 openshift-github-bot 2015-11-23 09:07:59 EST
Commits pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/1b71e4e1537a5bed7f441e4f223054a769f8b8ea
Bug 1277046 - fixed tagging ImageStreamImage from the same ImageStream
to point to original pull spec instead of the internal registry.

https://github.com/openshift/origin/commit/1e380fb9f4ff32c3da65d9efbbe525430d24e1ed
Merge pull request #5931 from soltysh/bug1277046

Merged by openshift-bot
Comment 10 Yan Du 2015-11-25 00:45:00 EST
Test with latest origin code (devenv_rhel7_2794), issue have been fixed. 

[root@Daphne origin]# oc tag recreate-example:v2 recreate-example:latest
Tag recreate-example:latest set to recreate-example@sha256:1318f08b141aa6a4cdca8c09fe8754b6c9f7802f8fc24e4e39ebf93e9d58472b.

[root@Daphne origin]# oc get pod
NAME                       READY     STATUS    RESTARTS   AGE
recreate-example-2-c7rix   1/1       Running   0          53s
recreate-example-2-fevtb   1/1       Running   0          53s

Move to verified.

Note You need to log in before you can comment on or make changes to this bug.