Red Hat Bugzilla – Bug 1442855
layers have zero size in imagestreams
Last modified: 2017-10-25 09:00:48 EDT
Description of problem: After creating an ImageStream object that references an image in registry.access.redhat.com layers are not pulled. Version-Release number of selected component (if applicable): v3.4.1.12 How reproducible: Always Steps to Reproduce: 1. oc cluster up --version v3.4 --image registry.access.redhat.com/openshift3/ose 2. oc create -f - << EOF { "kind": "ImageStream", "apiVersion": "v1", "metadata": { "name": "rhel7-test" }, "spec": { "dockerImageRepository": "registry.access.redhat.com/rhel7" } } EOF 3. Wait until tags are populated for imagestream rhel7-test 4. Check size of layers: oc get imagestreamtag lucarval-rhel7-test:latest -o json Actual results: .image.dockerImageLayers reports empty layers: [ { "name": "sha256:e50b1d9e90ba6803b2c3b8712a71299dafd614d8f8f6616bb692e2873207188f", "size": 0, "mediaType": "application/vnd.docker.container.image.rootfs.diff+x-gtar" }, { "name": "sha256:93b630859c0ea7dbe2b30ed22ccc5c53be542619405c6c4cb83cd2b5e7419648", "size": 0, "mediaType": "application/vnd.docker.container.image.rootfs.diff+x-gtar" } ] Expected results: Expect correct size to be set. Additional info: Using a different container registry to pull image from does not exhibit this issue. Tried "docker.io/centos" and "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7" (insecure registry), both seem to work properly. Issue is not seen when using an older version of OpenShift: "--version v3.4"
Typo in previous comment... It should read: Issue is not seen when using an older version of OpenShift: "--version v3.3"
I can actually see the problem with version 3.3 as well but only for last tag (latest is fine) rhel7-test:7.3 (same image as in latest) shows 0 size rhel7-test:7.2 is OK rhel7-test:7.0-21 is OK rhel7-test:7.0-23 is OK rhel7-test:latest is OK
Manifest must contain correct layer sizes - at least manifest V2 schema 2. It's not OpenShift's fault that registry.access.redhat.com serves bad data. This should be fixed on pulp's side or in our in build's infrastructure. I wonder why the docker client doesn't fail in pulling such an image. I guess it's satisfied with just matching digests. In OpenShift, we have a code that verifies/updates blob sizes if 0. Changing the code to handle 0 sizes for schema 2 would be trivial I guess. Nevertheless, I would consider it an low-priority bug for OpenShift if bug at all.
Keep in mind that size of 0 is merely a symptom. The real problem is that the image is actually not pulled from external registry into OpenShift's registry properly. Attempting to pull the image from OpenShift's internal registry causes an error: $ docker pull 172.30.1.1:5000/myproject/rhel7-test:latest Trying to pull repository 172.30.1.1:5000/myproject/rhel7-test ... manifest unknown: manifest unknown Further inspection shows that tags have been populated as expected, but something is up with the manifests: $ curl -H "Authorization: Bearer $(oc whoami -t)" -L http://172.30.1.1:5000/v2/myproject/rhel7-test/tags/list {"name":"myproject/rhel7-test","tags":["7.2","7.0-21","7.0-23","latest","7.3"]} $ curl -H "Authorization: Bearer $(oc whoami -t)" -L http://172.30.1.1:5000/v2/myproject/rhel7-test/manifests/latest {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Op":"Get","URL":"https://registry.access.redhat.com/v2/","Err":{"Op":"dial","Net":"tcp","Source":null,"Addr":null,"Err":{"Err":"read udp 172.17.0.4:59706-\u003e10.18.33.22:53: read: no route to host","Name":"registry.access.redhat.com","Server":"10.18.33.22:53","IsTimeout":false,"IsTemporary":false}}}}]}
This [1] and/or [2] should help. - https://docs.openshift.com/container-platform/3.4/install_config/http_proxies.html#proxying-docker-pull - https://docs.openshift.com/container-platform/3.4/install_config/http_proxies.html#configuring-hosts-for-proxies The proxy config option you mention is indeed something else which shouldn't be touched.
https://github.com/openshift/origin/pull/14166
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/5e2c3f3073501b2de7a192a3496f6ad7b9c24de7 Set layer size whether it found in cache or not Fixes bug 1442855 Signed-off-by: Oleg Bulatov <obulatov@redhat.com>
The code merged in Image: devenv-rhel7_6269 about 15 hours ago, and still could not find the image in aws:AMI.
Change to post status as no new puddles built out now
Let's use this track version 3.4.1 PR not merged: https://github.com/openshift/ose/pull/762
There is not new 3.4 puddle ready for verify this bug, we will verify it after new puddle ready.
Verified in OCP # openshift version openshift v3.4.1.37 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 1.# oc create -f /tmp/lg.yaml imagestream "rhel7-test" created 2.# oc get is NAME DOCKER REPO TAGS UPDATED rhel7-test registry.access.redhat.com/rhel7 latest,7.3,7.2 + 2 more... 5 seconds ago # oc get istag rhel7-test:latest -o json { "kind": "ImageStreamTag", "apiVersion": "v1", ................................ "dockerImageLayers": [ { "name": "sha256:4e5a7647df476dcb309aa02f6901239300e7103a914fd92acf540372c1dafe0c", "size": 72175335, "mediaType": "application/vnd.docker.container.image.rootfs.diff+x-gtar" }, { "name": "sha256:0001a3087112018853b83f67ffc311dab755d14393a69852d5e2f4aa01b35361", "size": 1194, "mediaType": "application/vnd.docker.container.image.rootfs.diff+x-gtar" } ], ......................
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3049