Description of problem: With OCP 3.3 registry configured to accept schema 2 and pull through turned on, and the same image is pushed into different two different projects, by different users, and no access to each others project. The ImageStream of the first pushed image points its location. However, the ImageStream of the second pushed image also references this location of the first pushed image. If the first project and image are deleted, the second project is now also (unknowingly) broken. Version-Release number of selected component (if applicable): How reproducible: Very reproducible using CDK and client environment Steps to Reproduce: Using CDK 2.3.0 1. docker pull nginx 2. vagrant up 3. vagrant ssh 4. sudo htpasswd -b /var/lib/openshift/openshift.local.config/master/user.htpasswd openshift-ops devel 5. exit 6. oc login -u admin -p admin 10.1.2.2:8443 7. oc set env dc/docker-registry -n default REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true 8. oc logout 9. oc login -u openshift-dev -p devel 10.1.2.2:8443 10. oc project sample-project 11. docker login -u openshift-dev -p `oc whoami -t` hub.openshift.rhel-cdk.10.1.2.2.xip.io 12. docker tag 01f818af747d hub.openshift.rhel-cdk.10.1.2.2.xip.io/sample-project/my-nginx:latest 13. docker push hub.openshift.rhel-cdk.10.1.2.2.xip.io/sample-project/my-nginx:latest 14. oc logout 15. oc login -u openshift-ops -p devel 10.1.2.2.:8443 16. oc new-project demo-project 17. docker login -u openshift-ops -p `oc whoami -t` hub.openshift.rhel-cdk.10.1.2.2.xip.io 18. docker tag 01f818af747d hub.openshift.rhel-cdk.10.1.2.2.xip.io/demo-project/my-nginx:latest 19. docker push hub.openshift.rhel-cdk.10.1.2.2.xip.io/demo-project/my-nginx:latest 20. oc describe is -n sample-project 21. oc describe is -n demo-project Actual results: Latest pushed image value when describing both ImageStreams will reference the image in the sample-project. Expected results: Additional info:
Created attachment 1235715 [details] Console output log from console output executing the steps to reproduce issue
This is what happens.. Manifest V2 schema 2 is content addressable. Neither image name nor tag affects resulting image id. Therefor the same image pushed to 2 different image streams will result in just one single image tagged in two different image streams. On the first push, dockerImageReference of the image is set to a pull spec of the first image stream. On the second push, the dockerImageReference isn't changed. Unfortunately, the image's reference is inherited by the image stream event [1] on tagging. Therefor the describe command shows the original reference. [1] https://github.com/miminar/origin/blob/master/pkg/image/registry/imagestreammapping/rest.go#L73 I'll fix that. > If the first project and image are deleted, the second project is now also (unknowingly) broken. This is actually expected. Once the one and only image object is deleted, it cannot be served from any image stream.
Here's a pull-request with the fix: https://github.com/openshift/origin/pull/12525 Note that the fix will make `oc describe is/<second-image-stream>` show proper dockerImageReference (e.g. 172.30.122.151:5000/demo-project/my-nginx@sha256:bef7440a8c5b7cc9993f483689e0606778f745d4dc253fd88d98554288dc3338). However, the original dockerImageReference will be still set on the image object. For example: $ oc describe -n demo-project isimage/my-nginx@sha256:bef7440a8c5b7cc9993f483689e0606778f745d4dc253fd88d98554288dc3338 Name: sha256:bef7440a8c5b7cc9993f483689e0606778f745d4dc253fd88d98554288dc3338 Namespace: <none> Created: 5 days ago Labels: <none> Annotations: openshift.io/image.managed=true Docker Image: 172.30.122.151:5000/sample-project/my-nginx@sha256:bef7440a8c5b7cc9993f483689e0606778f745d4dc253fd88d98554288dc3338 Image Name: sha256:bef7440a8c5b7cc9993f483689e0606778f745d4dc253fd88d98554288dc3338 But that's how it's supposed to work.
Is a back-port needed?
Confirmed with OCP3.3.1 , the issue has fixed: openshift version openshift v3.3.1.12 kubernetes v1.3.0+52492b4 etcd 2.3.0+git [root@ip-172-18-5-234 origin]# oc describe is/nginx-my -n zhouy1 Name: nginx-my Namespace: zhouy1 Created: About a minute ago Labels: <none> Annotations: <none> Docker Pull Spec: 172.30.117.229:5000/zhouy1/nginx-my Unique Images: 1 Tags: 1 latest pushed image * 172.30.117.229:5000/zhouy/my-nginx@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 About a minute ago [root@ip-172-18-5-234 origin]# oc describe is/my-nginx -n zhouy Name: my-nginx Namespace: zhouy Created: 5 minutes ago Labels: <none> Annotations: <none> Docker Pull Spec: 172.30.117.229:5000/zhouy/my-nginx Unique Images: 1 Tags: 1 latest pushed image * 172.30.117.229:5000/zhouy/my-nginx@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 5 minutes ago [root@ip-172-18-5-234 origin]# oc describe -n zhouy1 isimages/nginx-my@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 the server doesn't have a resource type "isimages" [root@ip-172-18-5-234 origin]# oc describe -n zhouy1 isimage/nginx-my@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Name: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Namespace: <none> Created: 6 minutes ago Labels: <none> Annotations: openshift.io/image.managed=true Docker Image: 172.30.117.229:5000/zhouy/my-nginx@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Image Name: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Image Size: 72.96 MB (first layer 194 B, last binary layer 52.47 MB) Image Created: 9 days ago Author: NGINX Docker Maintainers "docker-maint" Arch: amd64 Command: nginx -g daemon off; Working Dir: <none> User: <none> Exposes Ports: 443/tcp, 80/tcp Docker Labels: <none> Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin NGINX_VERSION=1.11.9-1~jessie [root@ip-172-18-5-234 origin]# oc delete project zhouy project "zhouy" deleted docker pull 172.30.117.229:5000/zhouy1/nginx-my Using default tag: latest Trying to pull repository 172.30.117.229:5000/zhouy1/nginx-my ... sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907: Pulling from 172.30.117.229:5000/zhouy1/nginx-my e12c678537ae: Pull complete 658824003923: Pull complete e633b39f284a: Pull complete Digest: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Status: Downloaded newer image for 172.30.117.229:5000/zhouy1/nginx-my:latest
@yinzhou I'm afraid the fix hasn't been back-ported to 3.3. It has been merged only in master. The verification steps are following: 1. configure registry to accept schema 2 [1] 2. push some image to the registry manually to image stream A 3. push the same image to the registry to different image stream B 4. ensure the resulting images have the same digest 5. describe both image streams 6. make sure that "Docker Image:" entry refers to the image stream being inspected, which means: $ oc describe -n ns is/A ... Docker Image: 172.30.122.151:5000/ns/A@sha256:xxxxxx ... $ oc describe -n ns is/B ... Docker Image: 172.30.122.151:5000/ns/B@sha256:xxxxxx ... [1] https://docs.openshift.org/latest/install_config/registry/extended_registry_configuration.html#middleware-overriding
Michal Minar: Thank you ! Have veryfied with ocp3.5, please see: openshift version openshift v3.5.0.18+9a5d1aa kubernetes v1.5.2+43a9be4 etcd 3.1.0 [root@zhouy testjson]# oc describe is ngix Name: ngix Namespace: testzy Created: Less than a second ago Labels: <none> Annotations: <none> Docker Pull Spec: 172.30.119.0:5000/testzy/ngix Unique Images: 1 Tags: 1 latest pushed image * 172.30.119.0:5000/testzy/ngix@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Less than a second ago [root@zhouy testjson]# oc describe is ngix1 -n zhouy Name: ngix1 Namespace: zhouy Created: Less than a second ago Labels: <none> Annotations: <none> Docker Pull Spec: 172.30.119.0:5000/zhouy/ngix1 Unique Images: 1 Tags: 1 latest pushed image * 172.30.119.0:5000/zhouy/ngix1@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Less than a second ago [root@zhouy testjson]# oc describe isimage/ngix@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Name: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Namespace: <none> Created: 35 seconds ago Labels: <none> Annotations: openshift.io/image.managed=true Docker Image: 172.30.119.0:5000/testzy/ngix@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Image Name: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Image Size: 72.96 MB (first layer 194 B, last binary layer 52.47 MB) Image Created: 2 weeks ago Author: NGINX Docker Maintainers "docker-maint" Arch: amd64 Command: nginx -g daemon off; Working Dir: <none> User: <none> Exposes Ports: 443/tcp, 80/tcp Docker Labels: <none> Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin NGINX_VERSION=1.11.9-1~jessie [root@zhouy testjson]# oc describe isimage/ngix1@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 -n zhouy Name: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Namespace: <none> Created: 42 seconds ago Labels: <none> Annotations: openshift.io/image.managed=true Docker Image: 172.30.119.0:5000/testzy/ngix@sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Image Name: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Image Size: 72.96 MB (first layer 194 B, last binary layer 52.47 MB) Image Created: 2 weeks ago Author: NGINX Docker Maintainers "docker-maint" Arch: amd64 Command: nginx -g daemon off; Working Dir: <none> User: <none> Exposes Ports: 443/tcp, 80/tcp Docker Labels: <none> Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin NGINX_VERSION=1.11.9-1~jessie [root@zhouy testjson]# oc delete project testzy project "testzy" deleted docker pull 172.30.119.0:5000/zhouy/ngix1:latest Trying to pull repository 172.30.119.0:5000/zhouy/ngix1 ... sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907: Pulling from 172.30.119.0:5000/zhouy/ngix1 e12c678537ae: Pull complete 658824003923: Pull complete e633b39f284a: Pull complete Digest: sha256:a5a0cadd0a8a9d2f42c0cd03a55cd0c610f6cc1466a6f9f9f96b4ae6513e9907 Status: Downloaded newer image for 172.30.119.0:5000/zhouy/ngix1:latest
Confirmed with OCP3.5 , the issue has fixed: [root@qe-yinzhou-master-1 ~]# openshift version openshift v3.5.0.40 kubernetes v1.5.2+43a9be4 etcd 3.1.0 [root@qe-yinzhou-master-1 ~]# oc get po NAME READY STATUS RESTARTS AGE docker-registry-4-bklfc 1/1 Running 0 29m registry-console-1-0m6v2 1/1 Running 0 1h router-1-z55vd 1/1 Running 0 1h [root@qe-yinzhou-master-1 ~]# oc env po docker-registry-4-bklfc --list # pods docker-registry-4-bklfc, container registry REGISTRY_HTTP_ADDR=:5000 REGISTRY_HTTP_NET=tcp REGISTRY_HTTP_SECRET=HKDi1dEu4CBATUcqC5S+/dTuE62mVIIIIl5Lup589rY= REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA=false REGISTRY_HTTP_TLS_KEY=/etc/secrets/registry.key REGISTRY_HTTP_TLS_CERTIFICATE=/etc/secrets/registry.crt REGISTRY_CONFIGURATION_PATH=/etc/registryconfig/config.yml REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true [root@qe-yinzhou-master-1 ~]# oc get is -n zhouy NAME DOCKER REPO TAGS UPDATED nginxzy 172.30.48.50:5000/zhouy/nginxzy latest 10 minutes ago [root@qe-yinzhou-master-1 ~]# oc describe is nginxzy -n zhouy Name: nginxzy Namespace: zhouy Created: 10 minutes ago Labels: <none> Annotations: <none> Docker Pull Spec: 172.30.48.50:5000/zhouy/nginxzy Unique Images: 1 Tags: 1 latest pushed image * 172.30.48.50:5000/zhouy/nginxzy@sha256:7f9c22c6b2ba14c609588a794fb40366c5d1eea2c70b8bb7474583994dab9e70 10 minutes ago [root@qe-yinzhou-master-1 ~]# oc get is -n zytest NAME DOCKER REPO TAGS UPDATED mynginx 172.30.48.50:5000/zytest/mynginx latest 8 minutes ago [root@qe-yinzhou-master-1 ~]# oc describe is mynginx -n zytest Name: mynginx Namespace: zytest Created: 8 minutes ago Labels: <none> Annotations: <none> Docker Pull Spec: 172.30.48.50:5000/zytest/mynginx Unique Images: 1 Tags: 1 latest pushed image * 172.30.48.50:5000/zytest/mynginx@sha256:7f9c22c6b2ba14c609588a794fb40366c5d1eea2c70b8bb7474583994dab9e70 8 minutes ago
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884