Created attachment 1668011 [details] image registry log Description of problem: Build fails because nodeshift/centos7-s2i-nodejs cannot be pulled from the public docker.io registry although imagecontentsourcepolicy is created after mirror the image to mirror registry. Version-Release number of selected component (if applicable): 4.4.0-0.nightly-2020-03-05-194612 How reproducible: always Steps to Reproduce: 1. Installed a disconnected env and create a project 2. Mirror extra image to mirror registry # oc image mirror docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9 wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/wewang/centos7-s2i-nodejs 3. Create icsp 4.Create secret of the mirror registry and link it to builder SA $oc create secret generic pullsecret --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson $oc secrets link builder pullsecret 5. Create a new build $oc new-build -D 'FROM docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9' --strategy=docker --to=test # oc logs -f build/test-1 Replaced Dockerfile FROM image docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9 Caching blobs under "/var/cache/blobs". Pulling image docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9 ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9" as "docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9": Error initializing source docker://nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9: error pinging docker registry registry-1.docker.io: Get https://registry-1.docker.io/v2/: dial tcp 54.174.39.59:443: i/o timeout Actual results: Failed to pull images Expected results: Should pull images successfully Additional info: Check registries.conf from nodes, already added the mirror info sh-4.4# more registries.conf unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] [[registry]] prefix = "" location = "docker.io/nodeshift/centos7-s2i-nodejs" mirror-by-digest-only = true [[registry.mirror]] location = "wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/nodeshift/centos7-s2i-nodejs" [[registry.mirror]] location = "wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openshift/centos7-s2i-nodejs" [[registry.mirror]] location = "wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/wewang/centos7-s2i-nodejs" [[registry]] prefix = "" location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev" mirror-by-digest-only = true [[registry.mirror]] location = "wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release" [[registry]] prefix = "" location = "registry.svc.ci.openshift.org/ocp/release" mirror-by-digest-only = true [[registry.mirror]] location = "wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release"
Turning up the logs to loglevel 6 revealed that the mirror was being used correctly, but the pull secret for the mirror registry was not found. As a result, buildah fell back to try pulling from all mirrors, then the upstream docker.io registry: ``` time="2020-03-06T18:41:14Z" level=debug msg="reference rewritten from 'docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9' to 'wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9'" time="2020-03-06T18:41:14Z" level=debug msg="reference rewritten from 'docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9' to 'wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9'" time="2020-03-06T18:41:14Z" level=debug msg="reference rewritten from 'docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9' to 'wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/wewang/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9'" time="2020-03-06T18:41:14Z" level=debug msg="reference rewritten from 'docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9' to 'docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9'" time="2020-03-06T18:41:14Z" level=debug msg="Trying to pull \"wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9\"" time="2020-03-06T18:41:14Z" level=debug msg="Credentials not found" time="2020-03-06T18:41:14Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration" time="2020-03-06T18:41:14Z" level=debug msg=" No signature storage configuration found for wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9" time="2020-03-06T18:41:14Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000" time="2020-03-06T18:41:14Z" level=debug msg=" crt: /etc/docker/certs.d/wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ca.crt" time="2020-03-06T18:41:14Z" level=debug msg="GET https://wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/" time="2020-03-06T18:41:14Z" level=debug msg="Ping https://wewang.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/ status 401" ... time="2020-03-06T18:41:14Z" level=debug msg="Trying to pull \"docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9\"" time="2020-03-06T18:41:14Z" level=debug msg="Credentials not found" time="2020-03-06T18:41:14Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration" time="2020-03-06T18:41:14Z" level=debug msg=" No signature storage configuration found for docker.io/nodeshift/centos7-s2i-nodejs@sha256:eea192da5dc21ddfbfbc1a1947ecb3c73e074e2d9516e5bed7ce66015464cce9" time="2020-03-06T18:41:14Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io" time="2020-03-06T18:41:14Z" level=debug msg="GET https://registry-1.docker.io/v2/" time="2020-03-06T18:41:44Z" level=debug msg="Ping https://registry-1.docker.io/v2/ err Get https://registry-1.docker.io/v2/: dial tcp 34.201.196.144:443: i/o timeout (&url.Error{Op:\"Get\", URL:\"https://registry-1.docker.io/v2/\", Err:(*net.OpError)(0xc0007e9a40)})" time="2020-03-06T18:41:44Z" level=debug msg="GET https://registry-1.docker.io/v1/_ping" time="2020-03-06T18:42:14Z" level=debug msg="Ping https://registry-1.docker.io/v1/_ping err Get https://registry-1.docker.io/v1/_ping: dial tcp 52.87.94.70:443: i/o timeout (&url.Error{Op:\"Get\", URL:\"https://registry-1.docker.io/v1/_ping\", Err:(*net.OpError)(0xc0000dda40)})" ``` I initially found that the mirror pull secret was not in the BuildConfig, nor was it linked to the builder service account. As a next step, I linked the mirror pull secret to the builder service account, and re-ran the build. It still failed to pull the image. Finally, I directly referenced the mirror pull secret as the pull secret for the build: ``` spec: ... strategy: dockerStrategy: from: kind: ImageStreamTag name: centos7-s2i-nodejs:latest pullSecret: name: mirrorsecret ``` This was successful. I suspect that this particular setup is causing us a bit of confusion. When using secrets from the service account: 1. When we first try to pull, we tell buildah to pull from docker.io and pass pull secrets for docker hub (if present) 2. Buildah then finds the mirror list and tries pulling from the mirrors 3. Buildah fails to pull from the mirrors because it does not have the auth credentials When a pull secret is specified in the BuildConfig, we assume that is the correct set of auth credentials and tell buildah to use those.
*** Bug 1810860 has been marked as a duplicate of this bug. ***
Tracking this in JIRA: https://issues.redhat.com/browse/RHDEVDOCS-2587 instead.
This work was completed on the attached Jira. Closing.