Description of problem: In disconnected cluster, cannot pull image to start a pod with jenkins which is enabled pullthrough(if disable pullthrough, pod can be running) jenkins-4-6bwsr 0/1 ImagePullBackOff 0 9m43s jenkins-4-deploy 1/1 Running 0 10m Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned wzheng/jenkins-4-6bwsr to xiuwang-dis-9pf69-compute-1 Normal Pulling 6m58s (x4 over 9m12s) kubelet, xiuwang-dis-9pf69-compute-1 Pulling image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" Warning Failed 6m43s (x4 over 8m57s) kubelet, xiuwang-dis-9pf69-compute-1 Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83": rpc error: code = Unknown desc = Error reading manifest sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83 in image-registry.openshift-image-registry.svc:5000/openshift/jenkins: unknown: unable to pull manifest from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Failed 6m43s (x4 over 8m57s) kubelet, xiuwang-dis-9pf69-compute-1 Error: ErrImagePull Warning Failed 6m31s (x6 over 8m56s) kubelet, xiuwang-dis-9pf69-compute-1 Error: ImagePullBackOff Normal BackOff 4m6s (x15 over 8m56s) kubelet, xiuwang-dis-9pf69-compute-1 Back-off pulling image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" Version-Release number of selected component (if applicable): 4.3.0-0.nightly-2019-12-29-173422 How reproducible: Always Steps to Reproduce: 1.Start a disconnected cluster 2.Double confirm jenkins imagestream is imported successfully 3.Create with jenkins template 4.Watch Jenkins pod Actual results: Cannot pull image to start jenkins. Expected results: Pod should be running Additional info: Met below error in registry pod log(detailed log is in attachment) time="2019-12-31T08:39:49.210362426Z" level=error msg="error getting secrets: <nil>" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=682231b9-4cf5-4be8-a491-199027edf24d http.request.method=GET http.request.remoteaddr="10.128.2.1:33490" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wzheng:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" time="2019-12-31T08:39:58.151797443Z" level=warning msg="error authorizing context: authorization header required" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=d7d45b7c-8efd-48d6-a170-c93062ec3c18 http.request.method=GET http.request.remoteaddr="10.128.2.1:33572" http.request.uri=/v2/ http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" time="2019-12-31T08:39:58.21106298Z" level=error msg="error getting secrets: <nil>" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=f7a23c2b-484f-427d-bcb7-1a56be1478df http.request.method=GET http.request.remoteaddr="10.128.2.1:33578" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wxj:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" time="2019-12-31T08:40:04.21096414Z" level=error msg="response completed with error" err.code="openshift pullthrough manifest" err.message="unable to pull manifest from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=682231b9-4cf5-4be8-a491-199027edf24d http.request.method=GET http.request.remoteaddr="10.128.2.1:33490" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=15.040250576s http.response.status=404 http.response.written=346 openshift.auth.user="system:serviceaccount:wzheng:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
The failing is not only for jenkins image, it's about the mirror image with digest id and pullthrough policy enabled. After add pull secret of quay.io to deployer sa under project, the image still failed to pull with pullthrough. $oc create secret generic pull --from-file=.dockerconfigjson=/home/wxj/.docker/config.json --type=kubernetes.io/dockerconfigjson $oc secrets link deployer pull --for=pull,mount
ImageContentSourcePolicy is not supported by the registry yet. https://issues.redhat.com/browse/DEVEXP-483
(In reply to Oleg Bulatov from comment #6) > ImageContentSourcePolicy is not supported by the registry yet. > > https://issues.redhat.com/browse/DEVEXP-483 If it is not supported, we may need to mention this in release note, since Jenkins imagestream is imported successfully and pullthrough is enabled by default.