Hide Forgot
Description of problem: Post cluster install the machineconfigs are going into degraded state because of authentication error. Version-Release number of selected component (if applicable): # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-02-22-074434 True False 114m Cluster version is 4.0.0-0.nightly-2019-02-22-074434 How reproducible: Always Steps to Reproduce: 1. Install a 4.0 cluster using the installer for version(4.0.0-0.nightly-2019-02-22-074434) 2. Inspect the machine configs # oc describe node | grep machineconfig | grep state machineconfiguration.openshift.io/state=Degraded machineconfiguration.openshift.io/state=Degraded machineconfiguration.openshift.io/state=Degraded machineconfiguration.openshift.io/state=Degraded machineconfiguration.openshift.io/state=Degraded machineconfiguration.openshift.io/state=Degraded Actual results: All the machineconfigs are in degraded state Expected results: All the machineconfigs should be in Done state Additional info: This was working fine till the following build (4.0.0-0.nightly-2019-02-18-223936) logs from the Daemon for the machine config shows authetication error - pivot.service: I0222 15:19:24.918784 4735 run.go:16] Running: skopeo inspect docker://registry.svc.ci.openshift.org/ocp/4.0-art-latest-2019-02-22-074434@sha256:660061d6eae3ee6d93ca836cd52e6033f1d611c629c1ce47cf272c9e9bda2488 pivot.service: time="2019-02-22T15:19:25Z" level=fatal msg="Error reading manifest sha256:660061d6eae3ee6d93ca836cd52e6033f1d611c629c1ce47cf272c9e9bda2488 in registry.svc.ci.openshift.org/ocp/4.0-art-latest-2019-02-22-074434: unauthorized: authentication required" pivot.service: W0222 15:19:25.393036 4735 run.go:45] skopeo failed: exit status 1; retrying... pivot.service: F0222 15:19:25.393066 4735 run.go:53] skopeo: timed out waiting for the condition E0222 15:19:25.399488 4690 daemon.go:435] Fatal error checking initial state of node: Checking initial state: Failed to run pivot: error queuing start job; got failed E0222 15:19:25.399509 4690 writer.go:90] Marking degraded due to: Checking initial state: Failed to run pivot: error queuing start job; got failed I0222 15:19:25.409135 4690 daemon.go:437] Entering degraded state; going to sleep
Moving to image registry since this seems to be an issue with the registry auth.
The repository 4.0-art-latest-2019-02-22-074434 has already been removed from the CI cluster. I guess the cluster has to be reinstalled. At least I don't know about any mechanisms that can upgrade images in the cluster.
But the question is why does the cluster try to use the `ocp` namespace? The problem is with the release image.
> The problem is with the release image. That would be the "Release" component then. But grabbing a recent promoted OCP nightly from [1], it looks like all the referenced images are in quay.io/openshift-release-dev/ocp-v4.0-art-dev: $ oc adm release info --pullspecs registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-05-045224 Name: 4.0.0-0.nightly-2019-03-05-045224 Digest: sha256:6db20d29d4ed8fe915d1a764b783997b23294663076f40dd04c0e29cdc972e4d Created: 2019-03-04 20:53:58 -0800 PST OS/Arch: linux/amd64 Manifests: 248 Release Metadata: Version: 4.0.0-0.nightly-2019-03-05-045224 Upgrades: <none> Component Versions: Kubernetes 1.12.4 Images: NAME PULL SPEC aws-machine-controllers quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:630e8118038ee97b8b3bbfed7d9b63e06c1346c606e11908064ea3f57bd9ff8e cli quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:93e16a8c56ec4031b5fa68683f75910aad57b54160a1e6054b3d3e96d9a4b376 cloud-credential-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbc8d586b2210ac44de554558fd299555e72fb662b6751589d69b173b03aa821 cluster-authentication-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04b2f8747268641e6ebaaa6f22e3f9462618042727d7ede0765d54152f6617ca cluster-autoscaler quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:708d70db17a329098dc27787ed56ca9169057f875a04f1351a4b84359519e2ac cluster-autoscaler-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf7b2bf274f9a1044d898636f805dd63f4a32d79ae6a578409fcbab6f80450e5 cluster-bootstrap quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20f69810b98dff7cba5c2ffe59ef826311c2bfd41ef989e2bd745acfb4ee6978 cluster-config-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dce94921d4b44bc73a30abf620a6f29b4791ad8345c6dfcccd2477fc626780cf cluster-dns-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:990949004540843c53df5beeebe699a1a657259f1325a290a6eba8397543398f cluster-image-registry-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a38e09e990a7054fd053e7083c7046b21419704c7f4ee49e7a1114ca3bfec37d cluster-ingress-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4dae4e56fcdf2d6e22c1d6903c387eeeb9943a04baa462596de5d20bace3f55 cluster-kube-apiserver-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6fde5fce4c7be626f282fbf8488c68348e714480a3e213cba00ca90f44deea16 cluster-kube-controller-manager-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f1041613b0387d0320465fd1a7bdc614b269797038bb3b4117d01b3e3a949ce cluster-kube-scheduler-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20532fd9805d7ddbc2c390f017da491a523cd49c5e56f83f2fc28214f8d7deac cluster-machine-approver quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9101f3fc7065fa1031f967452e0a3e9060009ff8db254ec9f580afd653946e8 cluster-monitoring-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2395b9b103ce60cfe22edbeb72b966a39decb8b66520dfb4c1641aacc5350bd cluster-network-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c304c3b772770e5d2625df991260fc18ae2ce15c1c8c5e11e775c84e5bcf900d cluster-node-tuned quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af39f34c4fb20f79aad7c7411831d71d98911d184650f420a135da527b766645 cluster-node-tuning-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c150270da0ebbe9a4dad837753d6bda987003c128b5b8e938ba94b68865ce63 cluster-openshift-apiserver-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71f521d9cb0b1f03fe78938f90df7a0a9bdf5a45a31c65b168533e507644da34 cluster-openshift-controller-manager-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:376af94fe741a0b2386e8d9e564fb062004504a4d6573ce625c5887e7b435ff2 cluster-samples-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c84a7d7c84f530467c67b0ec3c3347e55621199684f17fec6a47d12c7e5180a1 cluster-storage-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c925fe57d348c88842b9bf12fa546e3ec101e6b080bcb3bf224750e7f373946c cluster-svcat-apiserver-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6edd77ad4ef48f3b29045acc0c9a57b01dcfb2c42b497ab857fa4eb27d8a841c cluster-svcat-controller-manager-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9af8c72361b3a8bad34c44be6903e02c0f8a56b6ef4d459fce9c75ee38251093 cluster-version-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:576c8616aa750f406200ce866abe1944eda0cdba8f40cd00cf7f036981a9bbec configmap-reloader quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:191c84e2d90592c6b55735bd6887c95fa1fdafafd591d20e77872564370c4dd2 console quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:675da84ddba2b712c62b2fa37cbb57537d35b0c100df94a8f3f2bd970e12fab8 console-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83f955c8cb47bc3513c29dfa2adf0c115d471af15fb22854db6455f6b6689fb7 container-networking-plugins-supported quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bb8f1894bd1ddea7f09462d650389a0d4b522d4461425326453e8e196dce6a0 container-networking-plugins-unsupported quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1950dcd761d21cdacaa9dc6dc2fecbe3c2d201019bbf90bbd0fff47162cccac coredns quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:968463390b0d1f061f221258f42882595220e1ba9da44a0b706c4f12c832cdf8 deployer quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfeda3e706c1a91f5932dd4e26a22e0e71b79d281c2a1313f0598848ce1fda78 docker-builder quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:376588c8bca50cf1e521763463b5ddaabf8b35b4eb5989deecf28a31913ab8e6 docker-registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a457ab8bc3be66c97ce6babd77e6f1c1eaff91477bd5442460f927db06a15e55 etcd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:859c61bbf3a3b675f224389e98e0a4e781eb64d5232c6000e48b41ea3e2b1396 grafana quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:451711adc8314dbb4c7cd3b89b7362d4de8b0f07660b3a8cfc6962134071cf30 haproxy-router quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa68a227520b355f30771715dce4b19222e33e8f44de2391808c86abdbc45f9 hyperkube quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bb8e0bfeb6394e0bc1e7acad4761a6c5374694913bb0586bfef4f0591e08aad hypershift quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6714d2b2e3d9ab322f0cb547f24bc8f92b3859d92e50639d653f79c26c5504e0 installer quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f32a3c163878e31c7ea05d8e2c14e0244d143ae834d832fa9804e780fc73f28 jenkins quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44abe6ded17c081f1e68ecd692274c196f5c2aa6a58de712a20dff37f68f1f29 jenkins-agent-maven quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:94576dfae58f4f1b6a1c1d00e62ec3ff6e5a5d160ce4569523016bd67fb1e7e0 jenkins-agent-nodejs quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65394dd366283c9aecf0858fc4b4abfc67805609f88f6036c826ea68af2266b4 k8s-prometheus-adapter quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cf130687466e67ef549f7509d5eff5f6c52875101f461fbd1bd1809b5849927 kube-rbac-proxy quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:61180c69f1cba674625bd6c85995f524bdbdde27b2d08ea7efcecd7d3427409b kube-state-metrics quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:081a5fd32de4b763ba69dcb9681a81ba8a66ee1f11b5b245cd43dc569a3e8b73 libvirt-machine-controllers quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2298681b40e49478651dfe2e50fa82758a06e7b7661cfcda68c739ab34e5ef96 machine-api-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:008d807d5778280674f2e2de38ebaa36c8314f880bed5cebb79fd9bdb9783803 machine-config-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd23c28661f8e4b885bb52c485eb36f6a844e0e1d43fd19a30d602aed56f237d machine-config-daemon quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6527c163c19a84c5df242732ead867e2ac6fdf3b9a428fa4a26bc0be42a17c4d machine-config-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:caa659d13314b6171dc10476903b476b485f7481060ebda20ef036b6fcb25c95 machine-config-server quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96c72b03fd8dba3477ca3e6cbb962047e5d7572603dc9bdab41f17ef5c02877a machine-os-content quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:399582f711226ab1a0e76d8928ec55436dea9f8dc60976c10790d308b9d92181 multus-cni quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc30ffbcd1d4a9bc089ac11186d7871510f79c2be23987cc05fb34cc5e049a9c must-gather quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25696b1564b69149d9a00427a41d3d1f976a856d6c91dc7fd489aa6bc0a413c4 node quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd6c1aa279ea7c314c01ad479a9559672b9f89ab0fcaba077c7f0178014eb71b oauth-proxy quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b77f43952da38d5430266c73450318ccf5e494984f2ea56d1ed0a69a02661831 openstack-machine-controllers quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d00ac1df23f7fef606be396fda329a79a48617f45632490d974c5a55dd51b3e4 operator-lifecycle-manager quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9573b7412ab1d47f904867daa2bda7036ec2181d59fad3c6d4130f2d986498e8 operator-marketplace quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77893a9582ba7b8875ed2eb4cffdb0eb9734dd38eb3342fe70a21d569fb38f76 operator-registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fb9303766def228f753a9f158ddaf3443df1355be13f751bcc530660b8568b39 pod quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1541c84a201d757b650d1f70b13c132d09f4ee3cdc70b99b612ad90b71473c96 prom-label-proxy quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc8f3238c771fd67353c878ccd09d067f7da35b7675ed6b903ef5075ebf5980b prometheus quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e13b678e0ed933d4337831a61c4991b70cd1ce710b034025cd64b094866e64a prometheus-alertmanager quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd783c25ca1d3c0f6de1c9ed8c8397be2587c310bab1a2023d8e1f66207a52fe prometheus-config-reloader quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26c197ba809ac3cd18a79c1dde21812e97e6957e0dfc93ac6f4c50947f3dfe99 prometheus-node-exporter quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a101ab8ed3bade5d0699c3db99348ac70cedc7bceb551219b8e6c75b8723d21e prometheus-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16b754e943b93a3224712d9c3c9770a97a5b2f59444350c490a41ab13bd83c55 service-catalog quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f5d0e5b6d3801dad4e1f307875f691be0226ee01ebe547ed2992ef1e08aa4d78 service-serving-cert-signer quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce7b475d165d897c1a240b1a6d5096d7fa3a0e3f5a34940ccb61614a98970608 setup-etcd-environment quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:421b56a744c40871ddcc65cb022120e99cbfe15a5f2913602a7b2d35445f9b8a telemeter quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34b317b33e2fa8a97c77333c56397409be918c52e6db7ee347e087072f5d0f35 tests quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a038b58150c213f93f82a5439ff9a47a49d4dd90590461cd3aa0d91bb7af4301 Can you still reproduce? There may have been an issue with references to machine-os-content or something that have since been resolved. [1]: https://openshift-release.svc.ci.openshift.org/
The references to machine-os-content have been resolved. The configs are not going into degraded state anymore Version number: # ./openshift-install version ./openshift-install v4.0.16-1-dirty # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-06-074438 True False 31m Cluster version is 4.0.0-0.nightly-2019-03-06-074438 Steps to verify: 1. Install a 4.0 cluster using the installer for version(4.0.0-0.nightly-2019-03-06-074438) 2. Inspect the machine configs # oc describe node | grep machineconfig | grep state machineconfiguration.openshift.io/state: Done machineconfiguration.openshift.io/state: Done machineconfiguration.openshift.io/state: Done machineconfiguration.openshift.io/state: Done
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758