Bug 1787112 - [Disconnected]Meet "ImagePullBackOff" when try to deploy with mirror image and pullthrough enabled
Summary: [Disconnected]Meet "ImagePullBackOff" when try to deploy with mirror image an...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.3.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-31 08:53 UTC by Wenjing Zheng
Modified: 2020-07-02 03:58 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-03 12:51:08 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Wenjing Zheng 2019-12-31 08:53:29 UTC
Description of problem:
In disconnected cluster, cannot pull image to start a pod with jenkins which is enabled pullthrough(if disable pullthrough, pod can be running)
jenkins-4-6bwsr    0/1     ImagePullBackOff   0          9m43s
jenkins-4-deploy   1/1     Running            0          10m

Events:
  Type     Reason     Age                    From                                  Message
  ----     ------     ----                   ----                                  -------
  Normal   Scheduled  <unknown>              default-scheduler                     Successfully assigned wzheng/jenkins-4-6bwsr to xiuwang-dis-9pf69-compute-1
  Normal   Pulling    6m58s (x4 over 9m12s)  kubelet, xiuwang-dis-9pf69-compute-1  Pulling image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
  Warning  Failed     6m43s (x4 over 8m57s)  kubelet, xiuwang-dis-9pf69-compute-1  Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83": rpc error: code = Unknown desc = Error reading manifest sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83 in image-registry.openshift-image-registry.svc:5000/openshift/jenkins: unknown: unable to pull manifest from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     6m43s (x4 over 8m57s)  kubelet, xiuwang-dis-9pf69-compute-1  Error: ErrImagePull
  Warning  Failed     6m31s (x6 over 8m56s)  kubelet, xiuwang-dis-9pf69-compute-1  Error: ImagePullBackOff
  Normal   BackOff    4m6s (x15 over 8m56s)  kubelet, xiuwang-dis-9pf69-compute-1  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-29-173422

How reproducible:
Always

Steps to Reproduce:
1.Start a disconnected cluster
2.Double confirm jenkins imagestream is imported successfully
3.Create with jenkins template
4.Watch Jenkins pod

Actual results:
Cannot pull image to start jenkins.

Expected results:
Pod should be running

Additional info:
Met below error in registry pod log(detailed log is in attachment)
time="2019-12-31T08:39:49.210362426Z" level=error msg="error getting secrets: <nil>" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=682231b9-4cf5-4be8-a491-199027edf24d http.request.method=GET http.request.remoteaddr="10.128.2.1:33490" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wzheng:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
time="2019-12-31T08:39:58.151797443Z" level=warning msg="error authorizing context: authorization header required" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=d7d45b7c-8efd-48d6-a170-c93062ec3c18 http.request.method=GET http.request.remoteaddr="10.128.2.1:33572" http.request.uri=/v2/ http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64"
time="2019-12-31T08:39:58.21106298Z" level=error msg="error getting secrets: <nil>" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=f7a23c2b-484f-427d-bcb7-1a56be1478df http.request.method=GET http.request.remoteaddr="10.128.2.1:33578" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wxj:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"
time="2019-12-31T08:40:04.21096414Z" level=error msg="response completed with error" err.code="openshift pullthrough manifest" err.message="unable to pull manifest from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" go.version=go1.12.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=682231b9-4cf5-4be8-a491-199027edf24d http.request.method=GET http.request.remoteaddr="10.128.2.1:33490" http.request.uri="/v2/openshift/jenkins/manifests/sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83" http.request.useragent="cri-o/1.16.1-7.dev.rhaos4.3.gitcee3d66.el8 go/go1.13.4 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=15.040250576s http.response.status=404 http.response.written=346 openshift.auth.user="system:serviceaccount:wzheng:jenkins" vars.name=openshift/jenkins vars.reference="sha256:03d38ccf17b6b0b0490557c7516e1a468f6b21d080518f178c46e4333fa7ba83"

Comment 4 XiuJuan Wang 2020-01-02 03:38:04 UTC
The failing is not only for jenkins image, it's about the mirror image with digest id and pullthrough policy enabled.

After add pull secret of quay.io to deployer sa under project, the image still failed to pull with pullthrough.

$oc create secret generic pull --from-file=.dockerconfigjson=/home/wxj/.docker/config.json --type=kubernetes.io/dockerconfigjson
$oc secrets link deployer pull --for=pull,mount

Comment 6 Oleg Bulatov 2020-01-03 12:51:08 UTC
ImageContentSourcePolicy is not supported by the registry yet.

https://issues.redhat.com/browse/DEVEXP-483

Comment 7 Wenjing Zheng 2020-01-06 02:16:39 UTC
(In reply to Oleg Bulatov from comment #6)
> ImageContentSourcePolicy is not supported by the registry yet.
> 
> https://issues.redhat.com/browse/DEVEXP-483

If it is not supported, we may need to mention this in release note, since Jenkins imagestream is imported successfully and pullthrough is enabled by default.


Note You need to log in before you can comment on or make changes to this bug.