Bug 1787112

Summary: [Disconnected]Meet "ImagePullBackOff" when try to deploy with mirror image and pullthrough enabled
Product: OpenShift Container Platform Reporter: Wenjing Zheng <wzheng>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED DEFERRED QA Contact: Wenjing Zheng <wzheng>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.0CC: aos-bugs, shishika, wewang, wking, xiuwang
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-20 11:05:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Wenjing Zheng 2019-12-31 08:53:29 UTC 
Description of problem:
In disconnected cluster, cannot pull image to start a pod with jenkins which is enabled pullthrough(if disable pullthrough, pod can be running)
jenkins-4-6bwsr    0/1     ImagePullBackOff   0          9m43s
jenkins-4-deploy   1/1     Running            0          10m

Events:
  Type     Reason     Age                    From                                  Message
  ----     ------     ----                   ----                                  -------
  Normal   Scheduled  <unknown>              default-scheduler                     Successfully assigned wzheng/jenkins-4-6bwsr to xiuwang-dis-9pf69-compute-1
  Normal   Pulling    6m58s (x4 over 9m12s)  kubelet, xiuwang-dis-9pf69-compute-1  Pulling image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
  Warning  Failed     6m43s (x4 over 8m57s)  kubelet, xiuwang-dis-9pf69-compute-1  Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83": rpc error: code = Unknown desc = Error reading manifest sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83 in image-registry.openshift-image-registry.svc:5000/openshift/jenkins: unknown: unable to pull manifest from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     6m43s (x4 over 8m57s)  kubelet, xiuwang-dis-9pf69-compute-1  Error: ErrImagePull
  Warning  Failed     6m31s (x6 over 8m56s)  kubelet, xiuwang-dis-9pf69-compute-1  Error: ImagePullBackOff
  Normal   BackOff    4m6s (x15 over 8m56s)  kubelet, xiuwang-dis-9pf69-compute-1  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-29-173422

How reproducible:
Always

Steps to Reproduce:
1.Start a disconnected cluster
2.Double confirm jenkins imagestream is imported successfully
3.Create with jenkins template
4.Watch Jenkins pod

Actual results:
Cannot pull image to start jenkins.

Expected results:
Pod should be running

Additional info:
Met below error in registry pod log(detailed log is in attachment)
time="2019-12-31T08:39:49.210362426Z" level=error msg="error getting secrets: <nil>" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=682231b9-4cf5-4be8-a491-199027edf24d http.request.method=GET http.request.remoteaddr="10.128.2.1:33490" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wzheng:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
time="2019-12-31T08:39:58.151797443Z" level=warning msg="error authorizing context: authorization header required" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=d7d45b7c-8efd-48d6-a170-c93062ec3c18 http.request.method=GET http.request.remoteaddr="10.128.2.1:33572" http.request.uri=/v2/ http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64"
time="2019-12-31T08:39:58.21106298Z" level=error msg="error getting secrets: <nil>" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=f7a23c2b-484f-427d-bcb7-1a56be1478df http.request.method=GET http.request.remoteaddr="10.128.2.1:33578" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wxj:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
time="2019-12-31T08:40:04.21096414Z" level=error msg="response completed with error" err.code="openshift pullthrough manifest" err.message="unable to pull manifest from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=682231b9-4cf5-4be8-a491-199027edf24d http.request.method=GET http.request.remoteaddr="10.128.2.1:33490" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=15.040250576s http.response.status=404 http.response.written=346 openshift.auth.user="system:serviceaccount:wzheng:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
Comment 4 XiuJuan Wang 2020-01-02 03:38:04 UTC 
The failing is not only for jenkins image, it's about the mirror image with digest id and pullthrough policy enabled.

After add pull secret of quay.io to deployer sa under project, the image still failed to pull with pullthrough.

$oc create secret generic pull --from-file=.dockerconfigjson=/home/wxj/.docker/config.json --type=kubernetes.io/dockerconfigjson
$oc secrets link deployer pull --for=pull,mount
Comment 6 Oleg Bulatov 2020-01-03 12:51:08 UTC 
ImageContentSourcePolicy is not supported by the registry yet.

https://issues.redhat.com/browse/DEVEXP-483
Comment 7 Wenjing Zheng 2020-01-06 02:16:39 UTC 
(In reply to Oleg Bulatov from comment #6)
> ImageContentSourcePolicy is not supported by the registry yet.
> 
> https://issues.redhat.com/browse/DEVEXP-483

If it is not supported, we may need to mention this in release note, since Jenkins imagestream is imported successfully and pullthrough is enabled by default.