Bug 1713016
| Summary: | Bootstrap Node cannot pull images from CI registry without CI credentials | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Christian Hernandez <chernand> |
| Component: | Installer | Assignee: | Abhinav Dahiya <adahiya> |
| Installer sub component: | openshift-installer | QA Contact: | Johnny Liu <jialiu> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | bparees, nschuetz, sponnaga, wking |
| Version: | 4.1.0 | Keywords: | Reopened |
| Target Milestone: | --- | ||
| Target Release: | 4.1.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-05-22 19:57:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Christian Hernandez
2019-05-22 17:06:14 UTC
Here's what i did: ``` $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.1.0-rc.5.tar.gz | tar xzf - $ ./openshift-install version ./openshift-install v4.1.0-201905171742-dirty built from commit 6ba66dbb6c2c53e1901a6d167d1c813bbbf27f4d release image quay.io/openshift-release-dev/ocp-release@sha256:dc67ad5edd91ca48402309fe0629593e5ae3333435ef8d0bc52c2b62ca725021 $ cp aws-install-config.yaml dev/install-config.yaml $ AWS_PROFILE=openshift-dev ./bin/openshift-install --dir dev create cluster INFO Consuming "Install Config" from target directory INFO Creating infrastructure resources... INFO Waiting up to 30m0s for the Kubernetes API at https://api.adahiya-1.devcluster.openshift.com:6443... ... ## SSH on the bootstrap host to check progress $ ush core.172.239 Warning: Permanently added '54.89.172.239' (ECDSA) to the list of known hosts. Red Hat Enterprise Linux CoreOS 410.8.20190508.1 Beta WARNING: Direct SSH access to machines is not recommended. This node has been annotated with machineconfiguration.openshift.io/ssh=accessed --- This is the bootstrap node; it will be destroyed when the master is fully up. The primary service is "bootkube.service". To watch its status, run e.g. journalctl -b -f -u bootkube.service [core@ip-10-0-14-244 ~]$ journalctl -u bootkube -- Logs begin at Wed 2019-05-22 18:19:25 UTC, end at Wed 2019-05-22 18:22:03 UTC. -- May 22 18:20:21 ip-10-0-14-244 systemd[1]: Started Bootstrap a Kubernetes cluster. May 22 18:20:25 ip-10-0-14-244 bootkube.sh[1410]: Pulling release image... May 22 18:20:36 ip-10-0-14-244 bootkube.sh[1410]: 1705c1ec912d057906aaf92d30162c3d9da606e72464d4fa080638c2c8bfc173 May 22 18:20:55 ip-10-0-14-244 bootkube.sh[1410]: Rendering Cluster Version Operator Manifests... May 22 18:20:56 ip-10-0-14-244 bootkube.sh[1410]: Rendering cluster config manifests... May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_apiserver.cr> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_dns.crd.yaml May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_image.crd.ya> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_ingress.crd.> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_openshift-co> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_03_authorization-openshift_01_role> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_03_quota-openshift_01_clusterresou> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_03_security-openshift_01_scc.crd.y> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_authenticati> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_build.crd.ya> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_02_config.clust> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_console.crd.> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_featuregate.> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_oauth.crd.ya> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_openshift-co> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_project.crd.> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_infrastructu> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_network.crd.> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_scheduler.cr> May 22 18:20:58 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/config-bootstrap/manifests/0000_10_quota-openshift_01_clusterresou> May 22 18:20:59 ip-10-0-14-244 bootkube.sh[1410]: Rendering Kubernetes API server core manifests... May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/bootstrap-manifests/kube-apiserver-pod.ya> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/cluster-role-binding-kube-apise> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-csr-controller-ca.yaml May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-aggregator-client-signer> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/00_openshift-kube-apiserver-ns.> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/00_openshift-kube-apiserver-ope> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-sa-token-signing-cert> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-control-plane-client-sig> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-kube-apiserver-to-kubele> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-loadbalancer-serving-sig> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-localhost-serving-signer> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-service-network-serving-> May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/cluster-role-kube-apiserver.yaml May 22 18:21:02 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-admin-kubeconfig-clie> May 22 18:21:03 ip-10-0-14-244 bootkube.sh[1410]: Rendering Kubernetes Controller Manager core manifests... May 22 18:21:06 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-controller-manager-bootstrap/bootstrap-manifests/kube-control> May 22 18:21:06 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/secret-initial-kube-co> May 22 18:21:06 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/00_openshift-kube-cont> May 22 18:21:06 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/00_openshift-kube-cont> May 22 18:21:06 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/secret-csr-signer-sign> May 22 18:21:07 ip-10-0-14-244 bootkube.sh[1410]: Rendering Kubernetes Scheduler core manifests... May 22 18:21:09 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-scheduler-bootstrap/bootstrap-manifests/kube-scheduler-pod.ya> May 22 18:21:09 ip-10-0-14-244 bootkube.sh[1410]: Writing asset: /assets/kube-scheduler-bootstrap/manifests/00_openshift-kube-scheduler-ns.> May 22 18:21:10 ip-10-0-14-244 bootkube.sh[1410]: Rendering MCO manifests... May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.676621 1 bootstrap.go:86] Version: 4.1.0-201905171742-dirty (a3a9a27> May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.678780 1 bootstrap.go:141] manifests/machineconfigcontroller/control> May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.682465 1 bootstrap.go:141] manifests/master.machineconfigpool.yaml May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.682766 1 bootstrap.go:141] manifests/worker.machineconfigpool.yaml May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.683065 1 bootstrap.go:141] manifests/bootstrap-pod-v2.yaml May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.683495 1 bootstrap.go:141] manifests/machineconfigserver/csr-bootstr> May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: I0522 18:21:11.683847 1 bootstrap.go:141] manifests/machineconfigserver/kube-apiser> May 22 18:21:11 ip-10-0-14-244 bootkube.sh[1410]: Starting etcd certificate signer... May 22 18:21:13 ip-10-0-14-244 bootkube.sh[1410]: 6f4c11e7abe6093b4b33e90c0010486fba3f10c4d3462be842d77207014ef440 May 22 18:21:13 ip-10-0-14-244 bootkube.sh[1410]: Waiting for etcd cluster... ``` So if you had the correct pull-secret. I do not see errors when trying to bring up cluster esp. authenticated errors bootkube.service trying to pull from registry.svc.ci.openshift.org So it makes me think that reported has an Release Image Override setup. Can you attach the `.openshift_install.log` that will provide information on what release image was being used. Referenced Google Doc suggests overriding the release image, which is the problem. Stick to [1] and you should be fine (as Abhinav shows above). [1]: https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal/installing-bare-metal.html This is a bug since this happens when I DON"T override it. The output of the logfile isn't that helpful though ``` cat .openshift_install.log time="2019-05-16T06:46:34-07:00" level=debug msg="OpenShift Installer v4.1.0-201905091432-dirty" time="2019-05-16T06:46:34-07:00" level=debug msg="Built from commit 10c88f2455d26c1ac1465dbbc44825e4366eb467" time="2019-05-16T06:46:34-07:00" level=fatal msg="boostrap host address and at least one control plane host address must be provided" time="2019-05-16T06:47:07-07:00" level=debug msg="OpenShift Installer v4.1.0-201905091432-dirty" time="2019-05-16T06:47:07-07:00" level=debug msg="Built from commit 10c88f2455d26c1ac1465dbbc44825e4366eb467" time="2019-05-16T06:47:07-07:00" level=fatal msg="boostrap host address and at least one control plane host address must be provided" time="2019-05-16T06:47:24-07:00" level=debug msg="OpenShift Installer v4.1.0-201905091432-dirty" time="2019-05-16T06:47:24-07:00" level=debug msg="Built from commit 10c88f2455d26c1ac1465dbbc44825e4366eb467" time="2019-05-16T06:47:24-07:00" level=fatal msg="boostrap host address and at least one control plane host address must be provided" time="2019-05-16T06:47:25-07:00" level=debug msg="OpenShift Installer v4.1.0-201905091432-dirty" time="2019-05-16T06:47:25-07:00" level=debug msg="Built from commit 10c88f2455d26c1ac1465dbbc44825e4366eb467" time="2019-05-16T06:47:25-07:00" level=fatal msg="boostrap host address and at least one control plane host address must be provided" ``` Christian tried again with a fresh asset directory and got an .openshift_install.log which had: time="2019-05-22T12:04:38-07:00" level=debug msg="Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:dc67ad5edd91ca48402309fe0629593e5ae3333435ef8d0bc52c2b62ca725021" The bootstrap machine then attempted to pull that release instead of something from the CI-registry, so re-closing this. The current suspect for the earlier issues is contamination from leftover asset directory (docs about removing that in [1,2]). [1]: https://github.com/openshift/installer/blame/8811e63e3f70196f088d6bbf3993ca9043ac3909/README.md#L53-L55 [2]: https://github.com/openshift/openshift-docs/blame/enterprise-4.1/modules/installation-uninstall-aws.adoc#L42-L43 (this probably needs to be strengthened). (In reply to Christian Hernandez from comment #3) > This is a bug since this happens when I DON"T override it. > > The output of the logfile isn't that helpful though > > ``` > cat .openshift_install.log > time="2019-05-16T06:46:34-07:00" level=debug msg="OpenShift Installer > v4.1.0-201905091432-dirty" > time="2019-05-16T06:46:34-07:00" level=debug msg="Built from commit > 10c88f2455d26c1ac1465dbbc44825e4366eb467" > time="2019-05-16T06:46:34-07:00" level=fatal msg="boostrap host address and > at least one control plane host address must be provided" > time="2019-05-16T06:47:07-07:00" level=debug msg="OpenShift Installer > v4.1.0-201905091432-dirty" > time="2019-05-16T06:47:07-07:00" level=debug msg="Built from commit > 10c88f2455d26c1ac1465dbbc44825e4366eb467" > time="2019-05-16T06:47:07-07:00" level=fatal msg="boostrap host address and > at least one control plane host address must be provided" > time="2019-05-16T06:47:24-07:00" level=debug msg="OpenShift Installer > v4.1.0-201905091432-dirty" > time="2019-05-16T06:47:24-07:00" level=debug msg="Built from commit > 10c88f2455d26c1ac1465dbbc44825e4366eb467" > time="2019-05-16T06:47:24-07:00" level=fatal msg="boostrap host address and > at least one control plane host address must be provided" > time="2019-05-16T06:47:25-07:00" level=debug msg="OpenShift Installer > v4.1.0-201905091432-dirty" > time="2019-05-16T06:47:25-07:00" level=debug msg="Built from commit > 10c88f2455d26c1ac1465dbbc44825e4366eb467" > time="2019-05-16T06:47:25-07:00" level=fatal msg="boostrap host address and > at least one control plane host address must be provided" > ``` Is that the entire log file? Can you grep for `release image` in your `.openshift_install.log` please? It should say something like this: > $ grep 'release image' .openshift_install.log > time="2019-05-19T13:44:58-05:00" level=debug msg="Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:6f4cf2db7e63c4dba54496a72b83fec22c49293b520ff0cdb78f1e38b23f1ccb" |