Bug 1433232 - Can't pull image when the external repo of image changed
Summary: Can't pull image when the external repo of image changed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.5.z
Assignee: Oleg Bulatov
QA Contact: ge liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-17 06:47 UTC by zhou ying
Modified: 2017-06-15 18:37 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the registry takes dockerImageReference from an Image object Consequence: dockerImageReference is shared across ImageStreams, the same image is fetched using the same dockerImageReference for all ImageStreams Fix: dockerImageReference is taken from an ImageStream Result: the registry fetches the image from different places for different ImageStreams
Clone Of:
Environment:
Last Closed: 2017-06-15 18:37:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1425 0 normal SHIPPED_LIVE OpenShift Container Platform 3.5, 3.4, 3.3, and 3.2 bug fix update 2017-06-15 22:35:53 UTC

Description zhou ying 2017-03-17 06:47:58 UTC
Description of problem:
When the external repo of image changed, but the digest of image not change, in OpenShift the 'DOCKER REF' of the image still use the old one, then the pullthough function can't work .

Version-Release number of selected component (if applicable):
openshift version
openshift v3.5.0.54
kubernetes v1.5.2+43a9be4
etcd 3.1.0

How reproducible:
always

Steps to Reproduce:
1. Login openshift and create project;
2. Create a private registry in the user's project:
   `oc new-app --docker-image=registry:2`
3. Wait for the private registry running, push image to the private registry:
   `docker login -u user -p password  service_of_private_registry`
   `docker pull docker.io/busybox & docker tag docker.io/busybox service_of_private_registry/project/busybox:latest & docker push service_of_private_registry/project/busybox:latest`
4. Tag image from the private registry to local project:
   `oc tag service_of_private_registry/project/busybox:latest  myimagestream:latest`
5. Try to pull the imagestream, will succeed;
6. Delete the project and all the resource;
7. Create new project and repeat step2~ step5, the new imagestream will pull failed.


Actual results:
4. Check the image object, the 'DOCKER REF' of the image use the service of the private registry:
   `oc get images |grep 7685ee1d1b2b1c97ce73dc75248af94f1232288a0f8abe4edf8b3842b2294169`
sha256:7685ee1d1b2b1c97ce73dc75248af94f1232288a0f8abe4edf8b3842b2294169   172.30.49.41:5000/terga/busybox@sha256:7685ee1d1b2b1c97ce73dc75248af94f1232288a0f8abe4edf8b3842b2294169
7. The new imagestream pull failed with error: manifest unknown: manifest unknown

Expected results:
7. Should use the latest private registry service , and pull the imagestream succeed.

Additional info:
   Check the image object, still use the service of the old private registry, and logs from integrated docker-registry:
[root@host-8-175-119 origin]# oc get images |grep 7685ee1d1b2b1c97ce73dc75248af94f1232288a0f8abe4edf8b3842b2294169
sha256:7685ee1d1b2b1c97ce73dc75248af94f1232288a0f8abe4edf8b3842b2294169     172.30.49.41:5000/terga/busybox@sha256:7685ee1d1b2b1c97ce73dc75248af94f1232288a0f8abe4edf8b3842b2294169

time="2017-03-17T02:53:07.688650433Z"  level=error msg="error getting remote repository for image  \"172.30.49.41:5000/terga/busybox:latest\": Get http://172.30.49.41:5000/v2/:  dial tcp 172.30.49.41:5000: getsockopt: no route to host"  go.version=go1.7.4 http.request.host="172.30.69.20:5000"  http.request.id=817f1519-85c1-4cfd-8704-a1afc949b4cf  http.request.method=GET http.request.remoteaddr="10.128.0.1:37712"  http.request.uri="/v2/wf7d0/mystream/manifests/latest"  http.request.useragent="docker/1.12.6 go/go1.7.4  kernel/3.10.0-514.10.2.el7.x86_64 os/linux arch/amd64  UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"  instance.id=33a1698f-0f3b-45ef-9e28-08f4c6a304bb  openshift.auth.user=anonymous openshift.logger=registry  vars.name="wf7d0/mystream" vars.reference=latest 
time="2017-03-17T02:53:07.688787114Z"  level=error msg="response completed with error" err.code="manifest  unknown" err.detail="Get http://172.30.49.41:5000/v2/:  dial tcp 172.30.49.41:5000: getsockopt: no route to host"  err.message="manifest unknown" go.version=go1.7.4  http.request.host="172.30.69.20:5000"  http.request.id=817f1519-85c1-4cfd-8704-a1afc949b4cf  http.request.method=GET http.request.remoteaddr="10.128.0.1:37712"  http.request.uri="/v2/wf7d0/mystream/manifests/latest"  http.request.useragent="docker/1.12.6 go/go1.7.4  kernel/3.10.0-514.10.2.el7.x86_64 os/linux arch/amd64  UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"  http.response.contenttype="application/json; charset=utf-8"  http.response.duration=7.261142961s http.response.status=404  http.response.written=267  instance.id=33a1698f-0f3b-45ef-9e28-08f4c6a304bb  openshift.auth.user=anonymous openshift.logger=registry  vars.name="wf7d0/mystream" vars.reference=latest 
10.128.0.1  - - [17/Mar/2017:02:53:00 +0000] "GET  /v2/wf7d0/mystream/manifests/latest HTTP/1.1" 404 267 "" "docker/1.12.6  go/go1.7.4 kernel/3.10.0-514.10.2.el7.x86_64 os/linux arch/amd64  UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"

Comment 1 Wang Haoran 2017-04-20 10:13:45 UTC
Oleg,Could you please paste the pr here if you fixed this bug ?

Comment 2 Oleg Bulatov 2017-04-20 10:18:57 UTC
https://github.com/openshift/origin/pull/13639

Comment 3 openshift-github-bot 2017-05-02 20:50:49 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/3989f0401bb2acb14d434b3fbd919b1d032b4eb1
Use docker image reference from ImageStream

Fixes bug 1433232

Signed-off-by: Oleg Bulatov <obulatov>

Comment 4 zhou ying 2017-05-05 04:43:20 UTC
Will confirm this issue when new OCP puddle ready.

Comment 5 zhou ying 2017-05-08 07:23:15 UTC
The latest OCP openshift v3.5.5.13 , not merged with the PR.

Comment 9 zhou ying 2017-05-24 06:19:37 UTC
Confirmed with OCP3.5.5.18, the issue has fixed:
openshift version
openshift v3.5.5.18
kubernetes v1.5.2+43a9be4
etcd 3.1.0

[root@host-8-175-70 ~]# oc get images |grep busy
sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558   172.30.220.99:5000/busybox@sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558

[root@host-8-175-70 ~]# oc get is mystream -o yaml 
.....
status:
  dockerImageRepository: 172.30.68.104:5000/iz3vc/mystream
  tags:
  - items:
    - created: 2017-05-24T06:08:03Z
      dockerImageReference: 172.30.191.166:5000/busybox@sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558
      generation: 2
      image: sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558
    tag: latest

docker pull 172.30.68.104:5000/iz3vc/mystream:latest` @ssh://root.centralci.eng.rdu2.redhat.com
      Not logged in to 172.30.68.104:5000
      Trying to pull repository 172.30.68.104:5000/iz3vc/mystream ... 
      sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558: Pulling from 172.30.68.104:5000/iz3vc/mystream
      Digest: sha256:c79345819a6882c31b41bc771d9a94fc52872fa651b36771fbe0c8461d7ee558
      Status: Downloaded newer image for 172.30.68.104:5000/iz3vc/mystream:latest

Comment 11 errata-xmlrpc 2017-06-15 18:37:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1425


Note You need to log in before you can comment on or make changes to this bug.