Bug 1680103 - [4.0.0] Machineconfigs degraded because of authentication failure post install.
Summary: [4.0.0] Machineconfigs degraded because of authentication failure post install.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Tim Bielawa
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-22 17:30 UTC by Siva Reddy
Modified: 2019-06-04 10:44 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:44:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:44:33 UTC

Description Siva Reddy 2019-02-22 17:30:48 UTC
Description of problem:
   Post cluster install the machineconfigs are going into degraded state because of authentication error.

Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-02-22-074434   True        False         114m      Cluster version is 4.0.0-0.nightly-2019-02-22-074434

How reproducible:
Always

Steps to Reproduce:
1. Install a 4.0 cluster using the installer for version(4.0.0-0.nightly-2019-02-22-074434)
2. Inspect the machine configs
# oc describe node | grep machineconfig | grep state
    machineconfiguration.openshift.io/state=Degraded
    machineconfiguration.openshift.io/state=Degraded
    machineconfiguration.openshift.io/state=Degraded
    machineconfiguration.openshift.io/state=Degraded
    machineconfiguration.openshift.io/state=Degraded
    machineconfiguration.openshift.io/state=Degraded 

Actual results:
   All the machineconfigs are in degraded state

Expected results:
   All the machineconfigs should be in Done state

Additional info:
     This was working fine till the following build (4.0.0-0.nightly-2019-02-18-223936)

    logs from the Daemon for the machine config shows authetication error -

pivot.service: I0222 15:19:24.918784    4735 run.go:16] Running: skopeo inspect docker://registry.svc.ci.openshift.org/ocp/4.0-art-latest-2019-02-22-074434@sha256:660061d6eae3ee6d93ca836cd52e6033f1d611c629c1ce47cf272c9e9bda2488
pivot.service: time="2019-02-22T15:19:25Z" level=fatal msg="Error reading manifest sha256:660061d6eae3ee6d93ca836cd52e6033f1d611c629c1ce47cf272c9e9bda2488 in registry.svc.ci.openshift.org/ocp/4.0-art-latest-2019-02-22-074434: unauthorized: authentication required"
pivot.service: W0222 15:19:25.393036    4735 run.go:45] skopeo failed: exit status 1; retrying...
pivot.service: F0222 15:19:25.393066    4735 run.go:53] skopeo: timed out waiting for the condition
E0222 15:19:25.399488    4690 daemon.go:435] Fatal error checking initial state of node: Checking initial state: Failed to run pivot: error queuing start job; got failed
E0222 15:19:25.399509    4690 writer.go:90] Marking degraded due to: Checking initial state: Failed to run pivot: error queuing start job; got failed
I0222 15:19:25.409135    4690 daemon.go:437] Entering degraded state; going to sleep

Comment 1 Erica von Buelow 2019-02-28 16:04:53 UTC
Moving to image registry since this seems to be an issue with the registry auth.

Comment 2 Oleg Bulatov 2019-02-28 16:38:47 UTC
The repository 4.0-art-latest-2019-02-22-074434 has already been removed from the CI cluster. I guess the cluster has to be reinstalled. At least I don't know about any mechanisms that can upgrade images in the cluster.

Comment 3 Oleg Bulatov 2019-02-28 16:57:30 UTC
But the question is why does the cluster try to use the `ocp` namespace? The problem is with the release image.

Comment 4 W. Trevor King 2019-03-05 07:46:55 UTC
> The problem is with the release image.

That would be the "Release" component then.  But grabbing a recent promoted OCP nightly from [1], it looks like all the referenced images are in quay.io/openshift-release-dev/ocp-v4.0-art-dev:

$ oc adm release info --pullspecs registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-05-045224
Name:      4.0.0-0.nightly-2019-03-05-045224
Digest:    sha256:6db20d29d4ed8fe915d1a764b783997b23294663076f40dd04c0e29cdc972e4d
Created:   2019-03-04 20:53:58 -0800 PST
OS/Arch:   linux/amd64
Manifests: 248

Release Metadata:
  Version:  4.0.0-0.nightly-2019-03-05-045224
  Upgrades: <none>

Component Versions:
  Kubernetes 1.12.4

Images:
  NAME                                          PULL SPEC
  aws-machine-controllers                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:630e8118038ee97b8b3bbfed7d9b63e06c1346c606e11908064ea3f57bd9ff8e
  cli                                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:93e16a8c56ec4031b5fa68683f75910aad57b54160a1e6054b3d3e96d9a4b376
  cloud-credential-operator                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbc8d586b2210ac44de554558fd299555e72fb662b6751589d69b173b03aa821
  cluster-authentication-operator               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04b2f8747268641e6ebaaa6f22e3f9462618042727d7ede0765d54152f6617ca
  cluster-autoscaler                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:708d70db17a329098dc27787ed56ca9169057f875a04f1351a4b84359519e2ac
  cluster-autoscaler-operator                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf7b2bf274f9a1044d898636f805dd63f4a32d79ae6a578409fcbab6f80450e5
  cluster-bootstrap                             quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20f69810b98dff7cba5c2ffe59ef826311c2bfd41ef989e2bd745acfb4ee6978
  cluster-config-operator                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dce94921d4b44bc73a30abf620a6f29b4791ad8345c6dfcccd2477fc626780cf
  cluster-dns-operator                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:990949004540843c53df5beeebe699a1a657259f1325a290a6eba8397543398f
  cluster-image-registry-operator               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a38e09e990a7054fd053e7083c7046b21419704c7f4ee49e7a1114ca3bfec37d
  cluster-ingress-operator                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4dae4e56fcdf2d6e22c1d6903c387eeeb9943a04baa462596de5d20bace3f55
  cluster-kube-apiserver-operator               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6fde5fce4c7be626f282fbf8488c68348e714480a3e213cba00ca90f44deea16
  cluster-kube-controller-manager-operator      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f1041613b0387d0320465fd1a7bdc614b269797038bb3b4117d01b3e3a949ce
  cluster-kube-scheduler-operator               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20532fd9805d7ddbc2c390f017da491a523cd49c5e56f83f2fc28214f8d7deac
  cluster-machine-approver                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9101f3fc7065fa1031f967452e0a3e9060009ff8db254ec9f580afd653946e8
  cluster-monitoring-operator                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2395b9b103ce60cfe22edbeb72b966a39decb8b66520dfb4c1641aacc5350bd
  cluster-network-operator                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c304c3b772770e5d2625df991260fc18ae2ce15c1c8c5e11e775c84e5bcf900d
  cluster-node-tuned                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af39f34c4fb20f79aad7c7411831d71d98911d184650f420a135da527b766645
  cluster-node-tuning-operator                  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c150270da0ebbe9a4dad837753d6bda987003c128b5b8e938ba94b68865ce63
  cluster-openshift-apiserver-operator          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71f521d9cb0b1f03fe78938f90df7a0a9bdf5a45a31c65b168533e507644da34
  cluster-openshift-controller-manager-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:376af94fe741a0b2386e8d9e564fb062004504a4d6573ce625c5887e7b435ff2
  cluster-samples-operator                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c84a7d7c84f530467c67b0ec3c3347e55621199684f17fec6a47d12c7e5180a1
  cluster-storage-operator                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c925fe57d348c88842b9bf12fa546e3ec101e6b080bcb3bf224750e7f373946c
  cluster-svcat-apiserver-operator              quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6edd77ad4ef48f3b29045acc0c9a57b01dcfb2c42b497ab857fa4eb27d8a841c
  cluster-svcat-controller-manager-operator     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9af8c72361b3a8bad34c44be6903e02c0f8a56b6ef4d459fce9c75ee38251093
  cluster-version-operator                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:576c8616aa750f406200ce866abe1944eda0cdba8f40cd00cf7f036981a9bbec
  configmap-reloader                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:191c84e2d90592c6b55735bd6887c95fa1fdafafd591d20e77872564370c4dd2
  console                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:675da84ddba2b712c62b2fa37cbb57537d35b0c100df94a8f3f2bd970e12fab8
  console-operator                              quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83f955c8cb47bc3513c29dfa2adf0c115d471af15fb22854db6455f6b6689fb7
  container-networking-plugins-supported        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bb8f1894bd1ddea7f09462d650389a0d4b522d4461425326453e8e196dce6a0
  container-networking-plugins-unsupported      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1950dcd761d21cdacaa9dc6dc2fecbe3c2d201019bbf90bbd0fff47162cccac
  coredns                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:968463390b0d1f061f221258f42882595220e1ba9da44a0b706c4f12c832cdf8
  deployer                                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfeda3e706c1a91f5932dd4e26a22e0e71b79d281c2a1313f0598848ce1fda78
  docker-builder                                quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:376588c8bca50cf1e521763463b5ddaabf8b35b4eb5989deecf28a31913ab8e6
  docker-registry                               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a457ab8bc3be66c97ce6babd77e6f1c1eaff91477bd5442460f927db06a15e55
  etcd                                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:859c61bbf3a3b675f224389e98e0a4e781eb64d5232c6000e48b41ea3e2b1396
  grafana                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:451711adc8314dbb4c7cd3b89b7362d4de8b0f07660b3a8cfc6962134071cf30
  haproxy-router                                quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa68a227520b355f30771715dce4b19222e33e8f44de2391808c86abdbc45f9
  hyperkube                                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bb8e0bfeb6394e0bc1e7acad4761a6c5374694913bb0586bfef4f0591e08aad
  hypershift                                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6714d2b2e3d9ab322f0cb547f24bc8f92b3859d92e50639d653f79c26c5504e0
  installer                                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f32a3c163878e31c7ea05d8e2c14e0244d143ae834d832fa9804e780fc73f28
  jenkins                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44abe6ded17c081f1e68ecd692274c196f5c2aa6a58de712a20dff37f68f1f29
  jenkins-agent-maven                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:94576dfae58f4f1b6a1c1d00e62ec3ff6e5a5d160ce4569523016bd67fb1e7e0
  jenkins-agent-nodejs                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65394dd366283c9aecf0858fc4b4abfc67805609f88f6036c826ea68af2266b4
  k8s-prometheus-adapter                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cf130687466e67ef549f7509d5eff5f6c52875101f461fbd1bd1809b5849927
  kube-rbac-proxy                               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:61180c69f1cba674625bd6c85995f524bdbdde27b2d08ea7efcecd7d3427409b
  kube-state-metrics                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:081a5fd32de4b763ba69dcb9681a81ba8a66ee1f11b5b245cd43dc569a3e8b73
  libvirt-machine-controllers                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2298681b40e49478651dfe2e50fa82758a06e7b7661cfcda68c739ab34e5ef96
  machine-api-operator                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:008d807d5778280674f2e2de38ebaa36c8314f880bed5cebb79fd9bdb9783803
  machine-config-controller                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd23c28661f8e4b885bb52c485eb36f6a844e0e1d43fd19a30d602aed56f237d
  machine-config-daemon                         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6527c163c19a84c5df242732ead867e2ac6fdf3b9a428fa4a26bc0be42a17c4d
  machine-config-operator                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:caa659d13314b6171dc10476903b476b485f7481060ebda20ef036b6fcb25c95
  machine-config-server                         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96c72b03fd8dba3477ca3e6cbb962047e5d7572603dc9bdab41f17ef5c02877a
  machine-os-content                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:399582f711226ab1a0e76d8928ec55436dea9f8dc60976c10790d308b9d92181
  multus-cni                                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc30ffbcd1d4a9bc089ac11186d7871510f79c2be23987cc05fb34cc5e049a9c
  must-gather                                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25696b1564b69149d9a00427a41d3d1f976a856d6c91dc7fd489aa6bc0a413c4
  node                                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd6c1aa279ea7c314c01ad479a9559672b9f89ab0fcaba077c7f0178014eb71b
  oauth-proxy                                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b77f43952da38d5430266c73450318ccf5e494984f2ea56d1ed0a69a02661831
  openstack-machine-controllers                 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d00ac1df23f7fef606be396fda329a79a48617f45632490d974c5a55dd51b3e4
  operator-lifecycle-manager                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9573b7412ab1d47f904867daa2bda7036ec2181d59fad3c6d4130f2d986498e8
  operator-marketplace                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77893a9582ba7b8875ed2eb4cffdb0eb9734dd38eb3342fe70a21d569fb38f76
  operator-registry                             quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fb9303766def228f753a9f158ddaf3443df1355be13f751bcc530660b8568b39
  pod                                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1541c84a201d757b650d1f70b13c132d09f4ee3cdc70b99b612ad90b71473c96
  prom-label-proxy                              quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc8f3238c771fd67353c878ccd09d067f7da35b7675ed6b903ef5075ebf5980b
  prometheus                                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e13b678e0ed933d4337831a61c4991b70cd1ce710b034025cd64b094866e64a
  prometheus-alertmanager                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd783c25ca1d3c0f6de1c9ed8c8397be2587c310bab1a2023d8e1f66207a52fe
  prometheus-config-reloader                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26c197ba809ac3cd18a79c1dde21812e97e6957e0dfc93ac6f4c50947f3dfe99
  prometheus-node-exporter                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a101ab8ed3bade5d0699c3db99348ac70cedc7bceb551219b8e6c75b8723d21e
  prometheus-operator                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16b754e943b93a3224712d9c3c9770a97a5b2f59444350c490a41ab13bd83c55
  service-catalog                               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f5d0e5b6d3801dad4e1f307875f691be0226ee01ebe547ed2992ef1e08aa4d78
  service-serving-cert-signer                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce7b475d165d897c1a240b1a6d5096d7fa3a0e3f5a34940ccb61614a98970608
  setup-etcd-environment                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:421b56a744c40871ddcc65cb022120e99cbfe15a5f2913602a7b2d35445f9b8a
  telemeter                                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34b317b33e2fa8a97c77333c56397409be918c52e6db7ee347e087072f5d0f35
  tests                                         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a038b58150c213f93f82a5439ff9a47a49d4dd90590461cd3aa0d91bb7af4301

Can you still reproduce?  There may have been an issue with references to machine-os-content or something that have since been resolved.

[1]: https://openshift-release.svc.ci.openshift.org/

Comment 6 Siva Reddy 2019-03-12 15:37:24 UTC
The references to machine-os-content have been resolved. The configs are not going into degraded state anymore

Version number:
# ./openshift-install version
./openshift-install v4.0.16-1-dirty
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-06-074438   True        False         31m     Cluster version is 4.0.0-0.nightly-2019-03-06-074438


Steps to verify:
1. Install a 4.0 cluster using the installer for version(4.0.0-0.nightly-2019-03-06-074438)
2. Inspect the machine configs
# oc describe node | grep machineconfig | grep state
                    machineconfiguration.openshift.io/state: Done
                    machineconfiguration.openshift.io/state: Done
                    machineconfiguration.openshift.io/state: Done
                    machineconfiguration.openshift.io/state: Done

Comment 8 errata-xmlrpc 2019-06-04 10:44:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.