Bug 1737043 - [disconnected] failed to pull release image on bootstrap node.
Summary: [disconnected] failed to pull release image on bootstrap node.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.2.0
Assignee: Miloslav Trmač
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-02 12:30 UTC by Johnny Liu
Modified: 2019-11-05 05:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-05 14:23:53 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Johnny Liu 2019-08-02 12:30:00 UTC
Description of problem:

Version-Release number of the following components:
$ oc version
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-alpha.0-46-gd7b76974", GitCommit:"d7b76974f2100ac2722128f03cd9ee66d0a620d9", GitTreeState:"clean", BuildDate:"2019-08-01T15:23:47Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}

./openshift-install v4.2.0-201908010219-dirty
built from commit 1f8da8a771253e74db3bde6758acac2fdbfac0d3
release image registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682

release image:
registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-07-30-045028

rhcos version:
410.8.20190604.0

# rpm -qa|grep cri
criu-3.10-7.el8.x86_64
cri-tools-1.13.0-2.rhaos4.1.gitb69a0b9.el8.x86_64
subscription-manager-rhsm-certificates-1.23.8-35.el8.x86_64
cri-o-1.13.9-1.rhaos4.1.gitd70609a.el8.x86_64


How reproducible:
Always

Steps to Reproduce:
1. mirror release image to internal registry.
$ oc adm release mirror -a /home/installer2/mirror_pullsecret_config.json --from=registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-07-30-045028 --to=internal-registry.qe.devcluster.openshift.com:5000/ocp/release --to-release-image=internal-registry.qe.devcluster.openshift.com:5000/ocp/release:4.2.0-0.nightly-2019-07-30-045028
<--snip-->
info: Mirroring completed in 690ms (0B/s)
sha256:6c2c726a3a2ba85a721a6bad6ed4d2145cf28d134a91cb9cda027640a8a8902e internal-registry.qe.devcluster.openshift.com:5000/ocp/release:console-operator

Success
Update image:  internal-registry.qe.devcluster.openshift.com:5000/ocp/release:4.2.0-0.nightly-2019-07-30-045028
Mirror prefix: internal-registry.qe.devcluster.openshift.com:5000/ocp/release

To use the new mirrored repository to install, add the following section to the install-config.yaml:

imageContentSources:
- mirrors:
  - internal-registry.qe.devcluster.openshift.com:5000/ocp/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - internal-registry.qe.devcluster.openshift.com:5000/ocp/release
  source: registry.svc.ci.openshift.org/ocp/release


To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: example
spec:
  repositoryDigestMirrors:
  - mirrors:
    - internal-registry.qe.devcluster.openshift.com:5000/ocp/release
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  - mirrors:
    - internal-registry.qe.devcluster.openshift.com:5000/ocp/release
    source: registry.svc.ci.openshift.org/ocp/release

2. Modify install-config to add the following lines:
imageContentSources:
- mirrors:
  - internal-registry.qe.devcluster.openshift.com:5000/ocp/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - internal-registry.qe.devcluster.openshift.com:5000/ocp/release
  source: registry.svc.ci.openshift.org/ocp/release

3. trigger a upi install as common process, but the whole cluster have no interet connctivity.

Actual results:
Check bootkube log:
Aug 02 12:06:00 qe-jialiu-76bpt-bootstrap-0 systemd[1]: Started Bootstrap a Kubernetes cluster.
Aug 02 12:06:00 qe-jialiu-76bpt-bootstrap-0 bootkube.sh[1431]: Pulling release image...
Aug 02 12:06:00 qe-jialiu-76bpt-bootstrap-0 bootkube.sh[1431]: error pulling image "registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682": unable to pull registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: error loading registries: invalid URL: cannot be empty
Aug 02 12:06:00 qe-jialiu-76bpt-bootstrap-0 systemd[1]: bootkube.service: Main process exited, code=exited, status=125/n/a
Aug 02 12:06:00 qe-jialiu-76bpt-bootstrap-0 systemd[1]: bootkube.service: Failed with result 'exit-code'.

# cat /etc/containers/registries.conf 
[[registry]]
location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
insecure = false
mirror-by-digest-only = true

[[registry.mirror]]
location = "internal-registry.qe.devcluster.openshift.com:5000/ocp/release"
insecure = false


[[registry]]
location = "registry.svc.ci.openshift.org/ocp/release"
insecure = false
mirror-by-digest-only = true

[[registry.mirror]]
location = "internal-registry.qe.devcluster.openshift.com:5000/ocp/release"
insecure = false



Expected results:
bootkube should be able to pull release image from mirror registry successfully.

Additional info:
Try a new disconnected install, but switch bootimage to 42.80.20190801.1.
$ rpm -qa|grep cri
criu-3.10-7.el8.x86_64
cri-tools-1.14.0-1.rhaos4.2.el8.x86_64
subscription-manager-rhsm-certificates-1.23.8-35.el8.x86_64
cri-o-1.14.10-0.5.dev.rhaos4.2.gitcf4220b.el8.x86_64

Still failed. But error message is a bit different.
Aug 02 12:24:54 qe-jialiu1-7696m-bootstrap-0 systemd[1]: Started Bootstrap a Kubernetes cluster.
Aug 02 12:24:54 qe-jialiu1-7696m-bootstrap-0 bootkube.sh[31753]: Pulling release image...
Aug 02 12:25:54 qe-jialiu1-7696m-bootstrap-0 bootkube.sh[31753]: time="2019-08-02T12:25:54Z" level=error msg="Error pulling image ref //registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: Error initializing source docker://registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: pinging docker registry returned: Get https://registry.svc.ci.openshift.org/v2/: dial tcp 35.196.103.194:443: i/o timeout"
Aug 02 12:25:54 qe-jialiu1-7696m-bootstrap-0 bootkube.sh[31753]: Error: error pulling image "registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682": unable to pull registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/ocp/release@sha256:09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: pinging docker registry returned: Get https://registry.svc.ci.openshift.org/v2/: dial tcp 35.196.103.194:443: i/o timeout
Aug 02 12:25:54 qe-jialiu1-7696m-bootstrap-0 systemd[1]: bootkube.service: Main process exited, code=exited, status=125/n/a
Aug 02 12:25:54 qe-jialiu1-7696m-bootstrap-0 systemd[1]: bootkube.service: Failed with result 'exit-code'.

Comment 2 Clayton Coleman 2019-08-02 13:50:36 UTC
What version of podman is installed?

Comment 3 Johnny Liu 2019-08-02 14:06:41 UTC
In reply to Clayton Coleman from comment #2)
> What version of podman is installed?

In testing with 42.80.20190801.1 rhcos:
# rpm -qa|grep podman
podman-manpages-1.4.2-1.module+el8.1.0+3423+f0eda5e0.noarch
podman-1.4.2-1.module+el8.1.0+3423+f0eda5e0.x86_64


In testing with 410.8.20190604.0 rhcos:
# rpm -qa|grep podman
podman-1.0.2-1.dev.git96ccc2e.el8.x86_64

Comment 4 Seth Jennings 2019-08-02 14:21:31 UTC
I tested this an it worked for me.  Trying to figure out what the difference is.

Comment 5 Seth Jennings 2019-08-02 14:53:00 UTC
I just tried this sub'ing in my registry for the QE registry and it worked

$ oc adm release mirror --from=registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-07-30-045028 --to=registry.lab.variantweb.net/ocp/release --to-release-image=registry.lab.variantweb.net/ocp/release:4.2.0-0.nightly-2019-07-30-045028

install-config.yaml

imageContentSources:
- mirrors:
  - registry.lab.variantweb.net/ocp/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - registry.lab.variantweb.net/ocp/release
  source: registry.svc.ci.openshift.org/ocp/release

resulting registries.conf on the bootstrap node

[root@bootstrap ~]# cat /etc/containers/registries.conf 
[[registry]]
location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
insecure = false
mirror-by-digest-only = true

[[registry.mirror]]
location = "registry.lab.variantweb.net/ocp/release"
insecure = false


[[registry]]
location = "registry.svc.ci.openshift.org/ocp/release"
insecure = false
mirror-by-digest-only = true

[[registry.mirror]]
location = "registry.lab.variantweb.net/ocp/release"
insecure = false

$ journalctl -b -u bootkube.service | grep bootkube.sh
Aug 02 14:46:08 bootstrap bootkube.sh[1563]: Pulling release image...
Aug 02 14:46:15 bootstrap bootkube.sh[1563]: a85ba99003ad84d6a1fce72d7c476cb89c9aac6f245e6a3d8e773946a159cefd
Aug 02 14:46:33 bootstrap bootkube.sh[1563]: Rendering Cluster Version Operator Manifests...
Aug 02 14:46:41 bootstrap bootkube.sh[1563]: Rendering cluster config manifests...
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_infrastructure.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_02_config.clusterrole.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_quota-openshift_01_clusterresourcequota.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_oauth.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_03_security-openshift_01_scc.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_apiserver.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_build.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_dns.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_project.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_03_config-operator_01_proxy.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_03_quota-openshift_01_clusterresourcequota.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_authentication.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_image.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_ingress.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_network.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_openshift-config-managed-ns.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_openshift-config-ns.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_03_authorization-openshift_01_rolebindingrestriction.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_console.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_featuregate.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_imagecontentsourcepolicy.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_scheduler.crd.yaml
Aug 02 14:46:43 bootstrap bootkube.sh[1563]: Rendering Kubernetes API server core manifests...
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/bootstrap-manifests/kube-apiserver-pod.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-admin-kubeconfig-client-ca.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-aggregator-client-signer.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-control-plane-client-signer.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-kube-apiserver-to-kubelet-signer.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-loadbalancer-serving-signer.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/00_openshift-kube-apiserver-operator-ns.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/cluster-role-kube-apiserver.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-csr-controller-ca.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-sa-token-signing-certs.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-localhost-serving-signer.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-service-network-serving-signer.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/00_openshift-kube-apiserver-ns.yaml
Aug 02 14:46:45 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/cluster-role-binding-kube-apiserver.yaml
Aug 02 14:46:46 bootstrap bootkube.sh[1563]: Rendering Kubernetes Controller Manager core manifests...
Aug 02 14:46:48 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-controller-manager-bootstrap/bootstrap-manifests/kube-controller-manager-pod.yaml
Aug 02 14:46:48 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/00_openshift-kube-controller-manager-ns.yaml
Aug 02 14:46:48 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/00_openshift-kube-controller-manager-operator-ns.yaml
Aug 02 14:46:48 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/secret-csr-signer-signer.yaml
Aug 02 14:46:48 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/secret-initial-kube-controller-manager-service-account-private-key.yaml
Aug 02 14:46:49 bootstrap bootkube.sh[1563]: Rendering Kubernetes Scheduler core manifests...
Aug 02 14:46:51 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-scheduler-bootstrap/bootstrap-manifests/kube-scheduler-pod.yaml
Aug 02 14:46:51 bootstrap bootkube.sh[1563]: Writing asset: /assets/kube-scheduler-bootstrap/manifests/00_openshift-kube-scheduler-ns.yaml
Aug 02 14:46:52 bootstrap bootkube.sh[1563]: Rendering MCO manifests...
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.859940       1 bootstrap.go:86] Version: v4.2.0-201907291819-dirty (09c18e57cfa398653c3a55708702e6c962ab0fb3)
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.863805       1 bootstrap.go:177] manifests/machineconfigcontroller/controllerconfig.yaml
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.866496       1 bootstrap.go:177] manifests/master.machineconfigpool.yaml
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.866714       1 bootstrap.go:177] manifests/worker.machineconfigpool.yaml
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.866891       1 bootstrap.go:177] manifests/bootstrap-pod-v2.yaml
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.867135       1 bootstrap.go:177] manifests/machineconfigserver/csr-bootstrap-role-binding.yaml
Aug 02 14:46:55 bootstrap bootkube.sh[1563]: I0802 14:46:55.867335       1 bootstrap.go:177] manifests/machineconfigserver/kube-apiserver-serving-ca-configmap.yaml
Aug 02 14:46:56 bootstrap bootkube.sh[1563]: Starting etcd certificate signer...
Aug 02 14:46:58 bootstrap bootkube.sh[1563]: 9d39039a5e2f5f23c4199f082d84f2679be29e9df57bd36e10803ed66432cb5c
Aug 02 14:46:58 bootstrap bootkube.sh[1563]: Waiting for etcd cluster...

$ cat /etc/os-release | grep ^VERSION=
VERSION="42.80.20190801.1"

$ podman version
Version:            1.4.2
RemoteAPI Version:  1
Go Version:         go1.12.6
OS/Arch:            linux/amd64

Comment 6 Seth Jennings 2019-08-02 15:01:18 UTC
I do note that registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-07-30-045028 does not contain https://github.com/openshift/machine-config-operator/pull/1014

masters/worker bootstrapped with with bootstrap MCS will not get their /etc/containers/registries.conf set without it.

Comment 7 Seth Jennings 2019-08-02 16:58:09 UTC
Using 4.2.0-0.nightly-2019-08-01-113533 and RHCOS 42.80.20190801.1, I was able to fully install

Comment 8 Miloslav Trmač 2019-08-02 19:36:09 UTC
(In reply to Johnny Liu from comment #0)
> rhcos version:
> 410.8.20190604.0> Aug 02 12:06:00 qe-jialiu-76bpt-bootstrap-0 bootkube.sh[1431]: error pulling
> image
> "registry.svc.ci.openshift.org/ocp/release@sha256:
> 09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682": unable to
> pull
> registry.svc.ci.openshift.org/ocp/release@sha256:
> 09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: unable to
> pull image: Error initializing source
> docker://registry.svc.ci.openshift.org/ocp/release@sha256:
> 09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: error
> loading registries: invalid URL: cannot be empty> In testing with 410.8.20190604.0 rhcos:
> # rpm -qa|grep podman
> podman-1.0.2-1.dev.git96ccc2e.el8.x86_64

That is fairly old, and it uses a pre-release version of the registries.conf v2 format (using URL: instead of Location:). It will need to be updated to at the very least 1.4.0, preferably at least 1.4.1 (for containers/image 2.0.0).

Comment 9 Miloslav Trmač 2019-08-02 19:44:57 UTC
(In reply to Miloslav Trmač from comment #8)
> That is fairly old, and it uses a pre-release version of the registries.conf
> v2 format (using URL: instead of Location:). It will need to be updated to
> at the very least 1.4.0, preferably at least 1.4.1 (for containers/image
> 2.0.0).

… of course, I have overlooked the date in 410.8.20190604.0 . Do we _have_ to use / support such an old build? (We would probably just detect it and refuse to accept the mirror config, I guess.)

Comment 10 Miloslav Trmač 2019-08-02 20:01:14 UTC
(In reply to Johnny Liu from comment #0)
> Try a new disconnected install, but switch bootimage to 42.80.20190801.1.
> Aug 02 12:25:54 qe-jialiu1-7696m-bootstrap-0 bootkube.sh[31753]:
> time="2019-08-02T12:25:54Z" level=error msg="Error pulling image ref
> //registry.svc.ci.openshift.org/ocp/release@sha256:
> 09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: Error
> initializing source
> docker://registry.svc.ci.openshift.org/ocp/release@sha256:
> 09aa64d6f8700d59edd3585e32205c9698428e025f7e24ea1741ede3b2048682: pinging
> docker registry returned: Get https://registry.svc.ci.openshift.org/v2/:
> dial tcp 35.196.103.194:443: i/o timeout"

I’m afraid the current code only reports the error contacting the primary endpoint, failures to access the mirrors (if any) are not returned to the caller (and might not even be in the debug log). I have filed https://github.com/containers/image/issues/674 about that.

Still, a possible step to diagnose this would be to run (podman --log-level=debug pull docker://$the_image) and see if it reports anything useful about the mirror.

Comment 11 Johnny Liu 2019-08-05 07:29:20 UTC
(In reply to Seth Jennings from comment #6)
> I do note that
> registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-07-30-045028
> does not contain
> https://github.com/openshift/machine-config-operator/pull/1014
> 
> masters/worker bootstrapped with with bootstrap MCS will not get their
> /etc/containers/registries.conf set without it.

This probably is the root cause.
After I re-run testing using rhcos-42.80.20190801.1 + 4.2.0-0.nightly-2019-08-01-113533, it works well now.

Comment 12 W. Trevor King 2019-11-05 05:03:14 UTC
> That is fairly old, and it uses a pre-release version of the registries.conf v2 format (using URL: instead of Location:)

For posterity, the url -> location pivot happened in [1].

[1]: https://github.com/containers/image/pull/564/files#diff-a92cc839152361a483b38c88adae5bceL28-R32


Note You need to log in before you can comment on or make changes to this bug.