Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1815799

Summary: OpenShift console doesn't come up on KVM/libvirt/OCP 4.4 RC2
Product: OpenShift Container Platform Reporter: mchisine
Component: NetworkingAssignee: Andrew McDermott <amcdermo>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED WONTFIX Docs Contact:
Severity: low    
Priority: low CC: amcdermo, aos-bugs, jokerman, jshollen, mchisine, mfojtik, slaznick
Version: 4.4   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-18 19:50:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mchisine 2020-03-21 19:39:49 UTC 
Description of problem:


Version-Release number of selected component (if applicable):
OCP 4.4 RC2 on KVM/libvirt

How reproducible:


Steps to Reproduce:
With OCP 4.4 RC2 on KVM/libvirt, the console doesn't come up (https://github.com/marcredhat/upi/blob/master/ocp44.adoc)

Same instructions worked great with OCP 4.3: https://github.com/marcredhat/upi/blob/master/ocp43.adoc
Actual results:


Expected results:


Additional info:
Comment 1 mchisine 2020-03-22 05:16:07 UTC 
oc logs console-58c548674f-vwrgk
2020-03-21T19:28:52Z cmd/main: cookies are secure!
2020-03-21T19:28:53Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp44.local/oauth/token failed: Head https://oauth-openshift.apps.ocp44.local: EOF
Comment 2 bpeterse 2020-03-24 13:12:12 UTC 
Logs indicate the console is trying to come up but can't make contact with the OAuth server.  Either the OAuth server is not yet available, or there is a routing issue.  The console backend will retry several times before it exits because there is a dependency problem that will need to be resolved.

Can you provide the status objects for:
- console-operator (oc get console.operator.openshift.io -o yaml)
- authentication-operator (oc get authentication.operator.openshift.io -o yaml)

Logs for these pods may also be helpful:
- console-operator (oc log -f <console-operator-pod> -n openshift-console-operator)
- authentication-operator (oc log -f <authentication-operator-pod> -n openshift-authentication-operator)
Comment 3 mchisine 2020-03-24 21:38:06 UTC 
oc get console.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
  kind: Console
  metadata:
    annotations:
      release.openshift.io/create-only: "true"
    creationTimestamp: "2020-03-24T21:35:25Z"
    generation: 1
    name: cluster
    resourceVersion: "13264"
    selfLink: /apis/operator.openshift.io/v1/consoles/cluster
    uid: 603a51bb-49ff-4080-b9b7-f335c7570db3
  spec:
    managementState: Managed
  status:
    conditions:
    - lastTransitionTime: "2020-03-24T21:35:41Z"
      status: "False"
      type: ManagementStateDegraded
    - lastTransitionTime: "2020-03-24T21:35:41Z"
      reason: NoUnsupportedConfigOverrides
      status: "True"
      type: UnsupportedConfigOverridesUpgradeable
    - lastTransitionTime: "2020-03-24T21:35:41Z"
      status: "False"
      type: ResourceSyncControllerDegraded
    - lastTransitionTime: "2020-03-24T21:35:43Z"
      status: "False"
      type: ServiceSyncDegraded
    - lastTransitionTime: "2020-03-24T21:35:43Z"
      status: "False"
      type: ServiceSyncProgressing
    - lastTransitionTime: "2020-03-24T21:35:45Z"
      status: "False"
      type: RouteSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: RouteSyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: ConfigMapSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: ConfigMapSyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: ServiceCASyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: ServiceCASyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: TrustedCASyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: TrustedCASyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: CustomLogoSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: CustomLogoSyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: DefaultIngressCertValidationDegraded
    - lastTransitionTime: "2020-03-24T21:36:27Z"
      status: "False"
      type: DefaultIngressCertValidationProgressing
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: OAuthClientSecretSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: OAuthClientSecretSyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: OAuthClientSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: OAuthClientSyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: DeploymentSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: DeploymentSyncProgressing
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      message: Working toward version 4.4.0-rc.2
      reason: InProgress
      status: "True"
      type: SyncLoopRefreshProgressing
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      message: 0 pods available for console deployment
      reason: InsufficientReplicas
      status: "False"
      type: DeploymentAvailable
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "True"
      type: RouteAvailable
    generations:
    - group: apps
      hash: ""
      lastGeneration: 2
      name: console
      namespace: openshift-console
      resource: deployments
    observedGeneration: 1
    readyReplicas: 0
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Comment 4 mchisine 2020-03-24 21:39:02 UTC 
oc get authentication.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
  kind: Authentication
  metadata:
    annotations:
      release.openshift.io/create-only: "true"
    creationTimestamp: "2020-03-24T21:23:15Z"
    generation: 1
    name: cluster
    resourceVersion: "17485"
    selfLink: /apis/operator.openshift.io/v1/authentications/cluster
    uid: 3beb2e07-549a-483b-9c9a-3a66893402fb
  spec:
    managementState: Managed
  status:
    conditions:
    - lastTransitionTime: "2020-03-24T21:29:34Z"
      reason: NoUnsupportedConfigOverrides
      status: "True"
      type: UnsupportedConfigOverridesUpgradeable
    - lastTransitionTime: "2020-03-24T21:29:34Z"
      status: "False"
      type: ManagementStateDegraded
    - lastTransitionTime: "2020-03-24T21:36:20Z"
      reason: AsExpected
      status: "False"
      type: RouterCertsDegraded
    - lastTransitionTime: "2020-03-24T21:38:27Z"
      message: All 2 endpoints for oauth-server are reporting 'not ready'
      reason: NonReadyEndpoints
      status: "True"
      type: IngressStateEndpointsDegraded
    - lastTransitionTime: "2020-03-24T21:29:35Z"
      status: "False"
      type: IngressStatePodsDegraded
    - lastTransitionTime: "2020-03-24T21:29:35Z"
      status: "False"
      type: ResourceSyncControllerDegraded
    - lastTransitionTime: "2020-03-24T21:36:28Z"
      status: "False"
      type: RouteStatusDegraded
    - lastTransitionTime: "2020-03-24T21:35:25Z"
      status: "False"
      type: OperatorSyncDegraded
    - lastTransitionTime: "2020-03-24T21:36:33Z"
      status: "False"
      type: IdentityProviderConfigDegraded
    - lastTransitionTime: "2020-03-24T21:36:55Z"
      status: "False"
      type: RouteHealthDegraded
    - lastTransitionTime: "2020-03-24T21:36:55Z"
      status: "False"
      type: WellKnownEndpointDegraded
    - lastTransitionTime: "2020-03-24T21:36:55Z"
      message: got '404 Not Found' status while trying to GET the OAuth well-known
        https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint
        data
      reason: WellKnownNotReady
      status: "True"
      type: Progressing
    - lastTransitionTime: "2020-03-24T21:36:55Z"
      status: "False"
      type: Available
    generations:
    - group: apps
      hash: ""
      lastGeneration: 1
      name: oauth-openshift
      namespace: openshift-authentication
      resource: deployments
    observedGeneration: 1
    readyReplicas: 2
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Comment 5 mchisine 2020-03-24 21:41:38 UTC 
oc logs console-operator-6f55d4c7bd-dvd6

I0324 21:35:40.725554       1 cmd.go:191] Using service-serving-cert provided certificates
I0324 21:35:40.728566       1 observer_polling.go:155] Starting file observer
W0324 21:35:41.590640       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0324 21:35:41.590929       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
I0324 21:35:41.601303       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 21:35:41.601347       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 21:35:41.601390       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0324 21:35:41.601455       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0324 21:35:41.601783       1 dynamic_serving_content.go:129] Starting serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key
I0324 21:35:41.602469       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-console-operator.svc" [serving] validServingFor=[metrics.openshift-console-operator.svc,metrics.openshift-console-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1585085378" (2020-03-24 21:35:27 +0000 UTC to 2022-03-24 21:35:28 +0000 UTC (now=2020-03-24 21:35:41.602437571 +0000 UTC))
I0324 21:35:41.602952       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1585085741" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1585085741" (2020-03-24 20:35:40 +0000 UTC to 2021-03-24 20:35:40 +0000 UTC (now=2020-03-24 21:35:41.602933244 +0000 UTC))
I0324 21:35:41.603034       1 secure_serving.go:178] Serving securely on [::]:8443
I0324 21:35:41.603107       1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0324 21:35:41.611668       1 leaderelection.go:242] attempting to acquire leader lease  openshift-console-operator/console-operator-lock...
I0324 21:35:41.632203       1 leaderelection.go:252] successfully acquired lease openshift-console-operator/console-operator-lock
I0324 21:35:41.637657       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-console-operator", Name:"console-operator-lock", UID:"8a4b9578-9e86-4a95-8de6-3dccf8d71113", APIVersion:"v1", ResourceVersion:"10524", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 390cb69c-ec2a-4211-9387-27180ade7910 became leader
I0324 21:35:41.639894       1 resourcesync_controller.go:218] Starting ResourceSyncController
I0324 21:35:41.639978       1 status_controller.go:199] Starting StatusSyncer-console
I0324 21:35:41.640002       1 shared_informer.go:197] Waiting for caches to sync for LoggingSyncer
I0324 21:35:41.640015       1 management_state_controller.go:102] Starting management-state-controller-console
I0324 21:35:41.640027       1 shared_informer.go:197] Waiting for caches to sync for UnsupportedConfigOverridesController
I0324 21:35:41.640040       1 controller.go:128] starting ConsoleServiceSyncController
I0324 21:35:41.640052       1 controller.go:99] starting ConsoleResourceSyncDestinationController
I0324 21:35:41.640143       1 controller.go:53] Starting Console
I0324 21:35:41.701462       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 21:35:41.701620       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0324 21:35:41.701828       1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2020-03-24 21:15:45 +0000 UTC to 2020-03-25 21:15:45 +0000 UTC (now=2020-03-24 21:35:41.701811579 +0000 UTC))
I0324 21:35:41.702072       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-console-operator.svc" [serving] validServingFor=[metrics.openshift-console-operator.svc,metrics.openshift-console-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1585085378" (2020-03-24 21:35:27 +0000 UTC to 2022-03-24 21:35:28 +0000 UTC (now=2020-03-24 21:35:41.702060564 +0000 UTC))
I0324 21:35:41.702295       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1585085741" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1585085741" (2020-03-24 20:35:40 +0000 UTC to 2021-03-24 20:35:40 +0000 UTC (now=2020-03-24 21:35:41.70228124 +0000 UTC))
I0324 21:35:41.702719       1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "admin-kubeconfig-signer" [] issuer="<self>" (2020-03-24 21:15:42 +0000 UTC to 2030-03-22 21:15:42 +0000 UTC (now=2020-03-24 21:35:41.702708132 +0000 UTC))
I0324 21:35:41.702760       1 tlsconfig.go:157] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-signer" [] issuer="<self>" (2020-03-24 21:15:48 +0000 UTC to 2020-03-25 21:15:48 +0000 UTC (now=2020-03-24 21:35:41.702738627 +0000 UTC))
I0324 21:35:41.702776       1 tlsconfig.go:157] loaded client CA [2/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-control-plane-signer" [] issuer="<self>" (2020-03-24 21:15:48 +0000 UTC to 2021-03-24 21:15:48 +0000 UTC (now=2020-03-24 21:35:41.702769712 +0000 UTC))
I0324 21:35:41.702791       1 tlsconfig.go:157] loaded client CA [3/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-apiserver-to-kubelet-signer" [] issuer="<self>" (2020-03-24 21:15:48 +0000 UTC to 2021-03-24 21:15:48 +0000 UTC (now=2020-03-24 21:35:41.702784745 +0000 UTC))
I0324 21:35:41.702804       1 tlsconfig.go:157] loaded client CA [4/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="<self>" (2020-03-24 21:15:43 +0000 UTC to 2030-03-22 21:15:43 +0000 UTC (now=2020-03-24 21:35:41.702798327 +0000 UTC))
I0324 21:35:41.702819       1 tlsconfig.go:157] loaded client CA [5/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-csr-signer_@1585085374" [] issuer="kubelet-signer" (2020-03-24 21:29:34 +0000 UTC to 2020-03-25 21:15:48 +0000 UTC (now=2020-03-24 21:35:41.702812885 +0000 UTC))
I0324 21:35:41.702832       1 tlsconfig.go:157] loaded client CA [6/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2020-03-24 21:15:45 +0000 UTC to 2020-03-25 21:15:45 +0000 UTC (now=2020-03-24 21:35:41.70282635 +0000 UTC))
I0324 21:35:41.703042       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-console-operator.svc" [serving] validServingFor=[metrics.openshift-console-operator.svc,metrics.openshift-console-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1585085378" (2020-03-24 21:35:27 +0000 UTC to 2022-03-24 21:35:28 +0000 UTC (now=2020-03-24 21:35:41.703033428 +0000 UTC))
I0324 21:35:41.703248       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1585085741" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1585085741" (2020-03-24 20:35:40 +0000 UTC to 2021-03-24 20:35:40 +0000 UTC (now=2020-03-24 21:35:41.703229697 +0000 UTC))
I0324 21:35:41.740246       1 status_controller.go:120] clusteroperator/console not found
I0324 21:35:41.740573       1 shared_informer.go:204] Caches are synced for UnsupportedConfigOverridesController
I0324 21:35:41.740626       1 base_controller.go:45] Starting #1 worker of UnsupportedConfigOverridesController controller ...
I0324 21:35:41.740360       1 shared_informer.go:204] Caches are synced for LoggingSyncer
I0324 21:35:41.740776       1 base_controller.go:45] Starting #1 worker of LoggingSyncer controller ...
I0324 21:35:41.748674       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Upgradeable"}],"relatedObjects":[{"group":"operator.openshift.io","name":"cluster","resource":"consoles"},{"group":"config.openshift.io","name":"cluster","resource":"consoles"},{"group":"config.openshift.io","name":"cluster","resource":"infrastructures"},{"group":"config.openshift.io","name":"cluster","resource":"proxies"},{"group":"oauth.openshift.io","name":"console","resource":"oauthclients"},{"group":"","name":"openshift-console-operator","resource":"namespaces"},{"group":"","name":"openshift-console","resource":"namespaces"},{"group":"","name":"console-public","namespace":"openshift-config-managed","resource":"configmaps"}],"versions":[{"name":"operator","version":"4.4.0-rc.2"}]}}
I0324 21:35:41.748899       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/console version "operator" changed from "" to "4.4.0-rc.2"
I0324 21:35:41.759115       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Upgradeable"}],"relatedObjects":[{"group":"operator.openshift.io","name":"cluster","resource":"consoles"},{"group":"config.openshift.io","name":"cluster","resource":"consoles"},{"group":"config.openshift.io","name":"cluster","resource":"infrastructures"},{"group":"config.openshift.io","name":"cluster","resource":"proxies"},{"group":"oauth.openshift.io","name":"console","resource":"oauthclients"},{"group":"","name":"openshift-console-operator","resource":"namespaces"},{"group":"","name":"openshift-console","resource":"namespaces"},{"group":"","name":"console-public","namespace":"openshift-config-managed","resource":"configmaps"}],"versions":[{"name":"operator","version":"4.4.0-rc.2"}]}}
I0324 21:35:41.760670       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/console version "operator" changed from "" to "4.4.0-rc.2"
I0324 21:35:41.765312       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Upgradeable"}]}}
I0324 21:35:41.765396       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded set to False (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "consoles" "" "cluster"} {"config.openshift.io" "consoles" "" "cluster"} {"config.openshift.io" "infrastructures" "" "cluster"} {"config.openshift.io" "proxies" "" "cluster"} {"oauth.openshift.io" "oauthclients" "" "console"} {"" "namespaces" "" "openshift-console-operator"} {"" "namespaces" "" "openshift-console"} {"" "configmaps" "openshift-config-managed" "console-public"}]
E0324 21:35:41.766578       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:41.766601       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
I0324 21:35:41.768847       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ServiceCreated' Created Service/console -n openshift-console because it was missing
I0324 21:35:41.772348       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded changed from Unknown to False ("")
I0324 21:35:41.772628       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
E0324 21:35:41.863864       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
I0324 21:35:42.052383       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:42Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:35:42.052540       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Upgradeable changed from Unknown to True ("")
E0324 21:35:42.068142       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:42.068217       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
I0324 21:35:42.248216       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Upgradeable changed from Unknown to True ("")
E0324 21:35:42.254692       1 status.go:120] status update error: Operation cannot be fulfilled on consoles.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
E0324 21:35:42.647384       1 status.go:120] status update error: Operation cannot be fulfilled on consoles.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
E0324 21:35:42.647501       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:43.064985       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:43.065094       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:43.465920       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:43.871486       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:43.871625       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
I0324 21:35:44.049500       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:44Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:35:44.057986       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from Unknown to False ("")
E0324 21:35:44.247775       1 status.go:120] status update error: Operation cannot be fulfilled on consoles.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
E0324 21:35:44.247887       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:44.888286       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:44.888420       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:45.060899       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:45.461772       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:45.461872       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:45.653310       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
I0324 21:35:45.654816       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:45Z","message":"RouteSyncProgressing: route is not available at canonical host []","reason":"RouteSync_FailedHost","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:35:45.667417       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from False to True ("RouteSyncProgressing: route is not available at canonical host []")
E0324 21:35:46.066357       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:46.066476       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:46.261522       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:46.261679       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:46.858218       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:46.858314       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:47.060370       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:47.060437       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:47.466471       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:47.466603       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:47.665253       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:47.665323       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:48.058366       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:48.058467       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:48.264402       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:48.264557       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:48.464084       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:48.464179       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:48.676949       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:48.677098       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:49.979647       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:49.979792       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:50.636000       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:50.636028       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:52.571609       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:52.571729       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:35:56.114137       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:56.114169       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:35:57.710815       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:35:57.710915       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
E0324 21:36:06.370795       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:36:06.370864       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:36:07.968298       1 status.go:73] RouteSyncProgressing FailedHost route is not available at canonical host []
E0324 21:36:07.968428       1 controller.go:129] {Console Console} failed with: route is not available at canonical host []
I0324 21:36:19.365033       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/default-ingress-cert -n openshift-console because it was missing
E0324 21:36:26.875692       1 status.go:73] OCDownloadsSyncDegraded FailedCreate ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
E0324 21:36:26.875720       1 controller.go:280] clidownloads-sync-work-queue-key failed with : ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
I0324 21:36:27.640897       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/console-config -n openshift-console because it was missing
I0324 21:36:28.338964       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/console-oauth-config -n openshift-console because it was missing
E0324 21:36:28.511957       1 status.go:73] SyncLoopRefreshProgressing InProgress Changes made during sync updates, additional sync expected.
E0324 21:36:28.512060       1 status.go:73] DeploymentAvailable InsufficientReplicas 0 pods available for console deployment
I0324 21:36:28.513166       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeploymentCreated' Created Deployment.apps/console -n openshift-console because it was missing
I0324 21:36:28.641218       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/console-public -n openshift-config-managed:
cause by changes in data.consoleURL
I0324 21:36:28.676782       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:45Z","message":"SyncLoopRefreshProgressing: Changes made during sync updates, additional sync expected.","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:28Z","message":"DeploymentAvailable: 0 pods available for console deployment","reason":"Deployment_InsufficientReplicas","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:28.703066       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing message changed from "RouteSyncProgressing: route is not available at canonical host []" to "SyncLoopRefreshProgressing: Changes made during sync updates, additional sync expected.",Available changed from Unknown to False ("DeploymentAvailable: 0 pods available for console deployment")
E0324 21:36:32.036851       1 status.go:73] SyncLoopRefreshProgressing InProgress Changes made during sync updates, additional sync expected.
E0324 21:36:32.036872       1 status.go:73] DeploymentAvailable InsufficientReplicas 0 pods available for console deployment
I0324 21:36:32.037314       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
E0324 21:36:32.497767       1 status.go:73] SyncLoopRefreshProgressing InProgress Working toward version 4.4.0-rc.2
E0324 21:36:32.497789       1 status.go:73] DeploymentAvailable InsufficientReplicas 0 pods available for console deployment
I0324 21:36:32.637455       1 status_controller.go:176] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:35:45Z","message":"SyncLoopRefreshProgressing: Working toward version 4.4.0-rc.2","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:28Z","message":"DeploymentAvailable: 0 pods available for console deployment","reason":"Deployment_InsufficientReplicas","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:35:41Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:32.651956       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5f7133c9-4011-492e-ab26-e1a97b877095", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing message changed from "SyncLoopRefreshProgressing: Changes made during sync updates, additional sync expected." to "SyncLoopRefreshProgressing: Working toward version 4.4.0-rc.2"
E0324 21:36:34.455895       1 status.go:73] SyncLoopRefreshProgressing InProgress Working toward version 4.4.0-rc.2
E0324 21:36:34.456034       1 status.go:73] DeploymentAvailable InsufficientReplicas 0 pods available for console deployment
W0324 21:37:48.502703       1 reflector.go:326] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 77; INTERNAL_ERROR") has prevented the request from succeeding
W0324 21:37:48.502820       1 reflector.go:326] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: watch of *v1.OAuthClient ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 73; INTERNAL_ERROR") has prevented the request from succeeding
E0324 21:37:49.629850       1 status.go:73] SyncLoopRefreshProgressing InProgress Working toward version 4.4.0-rc.2
E0324 21:37:49.629879       1 status.go:73] DeploymentAvailable InsufficientReplicas 0 pods available for console deployment
E0324 21:37:50.986062       1 status.go:73] SyncLoopRefreshProgressing InProgress Working toward version 4.4.0-rc.2
E0324 21:37:50.986088       1 status.go:73] DeploymentAvailable InsufficientReplicas 0 pods available for console deployment
Comment 6 mchisine 2020-03-24 21:43:14 UTC 
oc logs authentication-operator-59757d4c57-npv7x

Copying system trust bundle
I0324 21:34:11.600700       1 cmd.go:191] Using service-serving-cert provided certificates
I0324 21:34:11.602845       1 observer_polling.go:155] Starting file observer
I0324 21:34:11.603887       1 observer_polling.go:155] Starting file observer
W0324 21:34:12.500932       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0324 21:34:12.501164       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
I0324 21:34:12.512357       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-authentication-operator.svc" [serving] validServingFor=[metrics.openshift-authentication-operator.svc,metrics.openshift-authentication-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1585085378" (2020-03-24 21:29:48 +0000 UTC to 2022-03-24 21:29:49 +0000 UTC (now=2020-03-24 21:34:12.512337788 +0000 UTC))
I0324 21:34:12.512495       1 leaderelection.go:242] attempting to acquire leader lease  openshift-authentication-operator/cluster-authentication-operator-lock...
I0324 21:34:12.512571       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1585085652" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1585085652" (2020-03-24 20:34:11 +0000 UTC to 2021-03-24 20:34:11 +0000 UTC (now=2020-03-24 21:34:12.512560701 +0000 UTC))
I0324 21:34:12.512600       1 secure_serving.go:178] Serving securely on [::]:8443
I0324 21:34:12.512640       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 21:34:12.512663       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 21:34:12.512686       1 dynamic_serving_content.go:129] Starting serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key
I0324 21:34:12.512711       1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0324 21:34:12.513067       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0324 21:34:12.513084       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0324 21:34:12.612905       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 21:34:12.613259       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0324 21:34:12.613408       1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2020-03-24 21:15:45 +0000 UTC to 2020-03-25 21:15:45 +0000 UTC (now=2020-03-24 21:34:12.613384185 +0000 UTC))
I0324 21:34:12.613728       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-authentication-operator.svc" [serving] validServingFor=[metrics.openshift-authentication-operator.svc,metrics.openshift-authentication-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1585085378" (2020-03-24 21:29:48 +0000 UTC to 2022-03-24 21:29:49 +0000 UTC (now=2020-03-24 21:34:12.613714552 +0000 UTC))
I0324 21:34:12.613971       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1585085652" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1585085652" (2020-03-24 20:34:11 +0000 UTC to 2021-03-24 20:34:11 +0000 UTC (now=2020-03-24 21:34:12.613960548 +0000 UTC))
I0324 21:34:12.614446       1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "admin-kubeconfig-signer" [] issuer="<self>" (2020-03-24 21:15:42 +0000 UTC to 2030-03-22 21:15:42 +0000 UTC (now=2020-03-24 21:34:12.6144365 +0000 UTC))
I0324 21:34:12.614474       1 tlsconfig.go:157] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-signer" [] issuer="<self>" (2020-03-24 21:15:48 +0000 UTC to 2020-03-25 21:15:48 +0000 UTC (now=2020-03-24 21:34:12.614466854 +0000 UTC))
I0324 21:34:12.614488       1 tlsconfig.go:157] loaded client CA [2/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-control-plane-signer" [] issuer="<self>" (2020-03-24 21:15:48 +0000 UTC to 2021-03-24 21:15:48 +0000 UTC (now=2020-03-24 21:34:12.614481969 +0000 UTC))
I0324 21:34:12.614507       1 tlsconfig.go:157] loaded client CA [3/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-apiserver-to-kubelet-signer" [] issuer="<self>" (2020-03-24 21:15:48 +0000 UTC to 2021-03-24 21:15:48 +0000 UTC (now=2020-03-24 21:34:12.614501283 +0000 UTC))
I0324 21:34:12.614538       1 tlsconfig.go:157] loaded client CA [4/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="<self>" (2020-03-24 21:15:43 +0000 UTC to 2030-03-22 21:15:43 +0000 UTC (now=2020-03-24 21:34:12.614531293 +0000 UTC))
I0324 21:34:12.614554       1 tlsconfig.go:157] loaded client CA [5/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-csr-signer_@1585085374" [] issuer="kubelet-signer" (2020-03-24 21:29:34 +0000 UTC to 2020-03-25 21:15:48 +0000 UTC (now=2020-03-24 21:34:12.614547932 +0000 UTC))
I0324 21:34:12.614576       1 tlsconfig.go:157] loaded client CA [6/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2020-03-24 21:15:45 +0000 UTC to 2020-03-25 21:15:45 +0000 UTC (now=2020-03-24 21:34:12.614569576 +0000 UTC))
I0324 21:34:12.614850       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-authentication-operator.svc" [serving] validServingFor=[metrics.openshift-authentication-operator.svc,metrics.openshift-authentication-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1585085378" (2020-03-24 21:29:48 +0000 UTC to 2022-03-24 21:29:49 +0000 UTC (now=2020-03-24 21:34:12.614840473 +0000 UTC))
I0324 21:34:12.615037       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1585085652" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1585085652" (2020-03-24 20:34:11 +0000 UTC to 2021-03-24 20:34:11 +0000 UTC (now=2020-03-24 21:34:12.615029287 +0000 UTC))
I0324 21:35:25.540153       1 leaderelection.go:252] successfully acquired lease openshift-authentication-operator/cluster-authentication-operator-lock
I0324 21:35:25.545286       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-authentication-operator", Name:"cluster-authentication-operator-lock", UID:"14793961-8b6b-4f77-9d5c-dbede1b17b2e", APIVersion:"v1", ResourceVersion:"10112", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 6743136a-a592-46a9-9879-e10654b48c1d became leader
I0324 21:35:25.562963       1 ingress_state_controller.go:144] Starting IngressStateController
I0324 21:35:25.570330       1 resourcesync_controller.go:218] Starting ResourceSyncController
I0324 21:35:25.570406       1 status_controller.go:199] Starting StatusSyncer-authentication
I0324 21:35:25.570449       1 unsupportedconfigoverrides_controller.go:152] Starting UnsupportedConfigOverridesController
I0324 21:35:25.570486       1 logging_controller.go:83] Starting LogLevelController
I0324 21:35:25.570521       1 controller.go:205] Starting RouterCertsDomainValidationController
I0324 21:35:25.570568       1 management_state_controller.go:102] Starting management-state-controller-authentication
I0324 21:35:25.570681       1 controller.go:53] Starting AuthenticationOperator2
I0324 21:35:25.570848       1 remove_stale_conditions.go:71] Starting RemoveStaleConditions
I0324 21:35:25.714593       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ocp44.local: []","reason":"IngressStateEndpoints_MissingSubsets::RouteStatus_FailedHost::RouterCerts_NoRouterCertSecret","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
E0324 21:35:25.715445       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
I0324 21:35:25.722837       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ocp44.local: []"
E0324 21:35:25.777983       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:25.976326       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:26.184300       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:26.401531       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:26.574346       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:26.793194       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:26.974668       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:27.175814       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:28.481420       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:31.056941       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:36.190831       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:46.460330       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:35:56.769111       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
W0324 21:36:06.775364       1 reflector.go:326] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 101; INTERNAL_ERROR") has prevented the request from succeeding
E0324 21:36:06.965011       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
E0324 21:36:07.809377       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
I0324 21:36:20.005709       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/v4-0-config-system-router-certs -n openshift-authentication because it was missing
E0324 21:36:20.028248       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
I0324 21:36:20.109522       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ocp44.local: []","reason":"IngressStateEndpoints_MissingSubsets::RouteStatus_FailedHost","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:20.122806       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ocp44.local: []" to "IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ocp44.local: []"
E0324 21:36:20.142371       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling the route: route is not available at canonical host oauth-openshift.apps.ocp44.local: []
I0324 21:36:28.181308       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/v4-0-config-system-metadata -n openshift-authentication because it was missing
I0324 21:36:28.200952       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/oauth-openshift -n openshift-config-managed because it was missing
I0324 21:36:28.307105       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ServiceCreated' Created Service/oauth-openshift -n openshift-authentication because it was missing
E0324 21:36:29.011328       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: failed handling service CA: config map has no service ca data: &v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"v4-0-config-system-service-ca", GenerateName:"", Namespace:"openshift-authentication", SelfLink:"/api/v1/namespaces/openshift-authentication/configmaps/v4-0-config-system-service-ca", UID:"29cf0bb3-2382-4aa5-9216-51ff8a446fa5", ResourceVersion:"12931", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63720682587, loc:(*time.Location)(0x3471900)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"oauth-openshift"}, Annotations:map[string]string{"service.alpha.openshift.io/inject-cabundle":"true"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Data:map[string]string(nil), BinaryData:map[string][]uint8(nil)}
I0324 21:36:29.012192       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server","reason":"IngressStateEndpoints_MissingSubsets","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:29.027976       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ocp44.local: []" to "IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
I0324 21:36:31.920456       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/v4-0-config-system-session -n openshift-authentication because it was missing
I0324 21:36:32.339849       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/v4-0-config-system-cliconfig -n openshift-authentication because it was missing
I0324 21:36:33.401149       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeploymentCreated' Created Deployment.apps/oauth-openshift -n openshift-authentication because it was missing
I0324 21:36:33.496659       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteHealthDegraded: failed to GET route: EOF","reason":"IngressStateEndpoints_MissingSubsets::RouteHealth_FailedGet","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
E0324 21:36:33.499374       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: EOF
I0324 21:36:33.522248       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteHealthDegraded: failed to GET route: EOF"
E0324 21:36:37.607980       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: EOF
E0324 21:36:40.829101       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: EOF
I0324 21:36:43.957638       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: All 1 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF","reason":"IngressStateEndpoints_NonReadyEndpoints::RouteHealth_FailedGet","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:43.979180       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRouteHealthDegraded: failed to GET route: EOF" to "IngressStateEndpointsDegraded: All 1 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF"
I0324 21:36:44.313737       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF","reason":"IngressStateEndpoints_NonReadyEndpoints::RouteHealth_FailedGet","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:44.322288       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 1 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF"
I0324 21:36:46.194732       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: All 1 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF","reason":"IngressStateEndpoints_NonReadyEndpoints::RouteHealth_FailedGet","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
E0324 21:36:46.206344       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: EOF
I0324 21:36:46.211164       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF" to "IngressStateEndpointsDegraded: All 1 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF"
E0324 21:36:49.350454       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: EOF
I0324 21:36:50.380130       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:32:21Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF","reason":"IngressStateEndpoints_NonReadyEndpoints::RouteHealth_FailedGet","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:50.400622       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 1 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nRouteHealthDegraded: failed to GET route: EOF"
I0324 21:36:50.438287       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"RouteHealthDegraded: failed to GET route: EOF","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"NoData","status":"Unknown","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:50.445873       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded changed from True to False ("RouteHealthDegraded: failed to GET route: EOF")
I0324 21:36:55.599206       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:36:55.616623       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "RouteHealthDegraded: failed to GET route: EOF" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'",Progressing changed from Unknown to True ("Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data"),Available changed from Unknown to False ("")
I0324 21:37:25.808650       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:37:25.822610       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'" to ""
I0324 21:37:28.006117       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:37:28.036635       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'"
W0324 21:37:49.136404       1 reflector.go:326] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 259; INTERNAL_ERROR") has prevented the request from succeeding
I0324 21:38:26.594316       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:38:26.603606       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'" to ""
I0324 21:38:27.996790       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:38:28.006607       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'"
I0324 21:39:26.792151       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:39:26.802835       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'" to ""
I0324 21:39:27.998465       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:39:28.011635       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'"
I0324 21:40:26.795185       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:40:26.806774       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'" to ""
I0324 21:40:28.008393       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:40:28.018540       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'"
I0324 21:40:57.201761       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.163:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:40:57.211078       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Progressing message changed from "Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint data" to "Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.163:6443/.well-known/oauth-authorization-server endpoint data"
I0324 21:41:26.795147       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.163:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:41:26.809333       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'" to ""
I0324 21:41:28.016944       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.163:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:41:28.042038       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'"
I0324 21:41:48.982897       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.983028       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.983283       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.983500       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.983687       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.983892       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.984281       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.984613       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.984869       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985047       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985089       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985275       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985292       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985447       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985280       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985288       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985692       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985765       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985877       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.985926       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.986037       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.986081       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.986198       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.986244       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:48.986383       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0324 21:41:51.452579       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.163:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:41:51.463694       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'" to ""
I0324 21:41:56.360058       1 status_controller.go:176] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-03-24T21:36:50Z","message":"IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-03-24T21:36:55Z","message":"Progressing: got '404 Not Found' status while trying to GET the OAuth well-known https://192.168.122.163:6443/.well-known/oauth-authorization-server endpoint data","reason":"_WellKnownNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-03-24T21:36:55Z","status":"False","type":"Available"},{"lastTransitionTime":"2020-03-24T21:29:34Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0324 21:41:56.371331       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"85f7e17e-d562-47d9-be59-f61ab9b8ca74", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'
Comment 7 mchisine 2020-03-24 21:45:20 UTC 
12m         Normal    OperatorStatusChanged   namespace/openshift-authentication-operator      Status for clusteroperator/authentication changed: Degraded changed from False to True ("RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server")
Comment 8 mchisine 2020-03-24 21:56:32 UTC 
FIXED as follows:
BASE_DOMAIN="$(oc get dns.config/cluster -o 'jsonpath={.spec.baseDomain}')"


INGRESS_DOMAIN="$(oc get ingress.config/cluster -o 'jsonpath={.spec.domain}')"


echo $BASE_DOMAIN
ocp44.local
echo $INGRESS_DOMAIN
apps.ocp44.local


    openssl genrsa -out example-ca.key 2048
    openssl req -x509 -new -key example-ca.key -out example-ca.crt -days 1 -subj "/C=US/ST=NC/L=Chocowinity/O=OS3/OU=Eng/CN=$BASE_DOMAIN"
    openssl genrsa -out example.key 2048
    openssl req -new -key example.key -out example.csr -subj "/C=US/ST=NC/L=Chocowinity/O=OS3/OU=Eng/CN=*.$INGRESS_DOMAIN"
    openssl x509 -req -in example.csr -CA example-ca.crt -CAkey example-ca.key -CAcreateserial -out example.crt -days 1




2. Configure the CA as the cluster proxy CA:

    oc -n openshift-config create configmap custom-ca --from-file=ca-bundle.crt=example-ca.crt
    oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

3. Configure the certificate as the ingresscontroller's default certificate:

    oc -n openshift-ingress create secret tls custom-default-cert --cert=example.crt --key=example.key
    oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"defaultCertificate":{"name":"custom-default-cert"}}}'

    oc -n openshift-config create configmap custom-ca --from-file=ca-bundle.crt=example-ca.crt
        oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'


  oc -n openshift-config create configmap custom-ca --from-file=ca-bundle.crt=example-ca.crt
            oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
Comment 9 mchisine 2020-03-24 22:44:13 UTC 
Updated step-by-step instructions / screenshots at https://github.com/marcredhat/upi/blob/master/ocp44.adoc
Comment 10 bpeterse 2020-03-27 19:52:24 UTC 
Console will not be ready if OAuth is not ready.
Authentication operator is indicating it is not ready.  There seems to be a trail leading further up the line, with issues like:
- route is not available at canonical host oauth-openshift.apps.ocp44.local
- lastTransitionTime: "2020-03-24T21:36:55Z"
      message: got '404 Not Found' status while trying to GET the OAuth well-known
        https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint
        data
      reason: WellKnownNotReady

Passing to auth for further investigation.
Comment 11 Standa Laznicka 2020-03-30 07:55:18 UTC 
oauth-proxy is something different, I believe you wanted api-auth component.

Also, if the report is correct, then the authn operator at the end reports:
    - lastTransitionTime: "2020-03-24T21:38:27Z"
      message: All 2 endpoints for oauth-server are reporting 'not ready'
      reason: NonReadyEndpoints
      status: "True"
      type: IngressStateEndpointsDegraded
    - lastTransitionTime: "2020-03-24T21:36:55Z"
      message: got '404 Not Found' status while trying to GET the OAuth well-known
        https://192.168.122.117:6443/.well-known/oauth-authorization-server endpoint
        data
      reason: WellKnownNotReady
      status: "True"
      type: Progressing

Something might be wrong with the control plane, but from the logs it's hard to tell whether it's nodes/networking/routing/controllers.

While the diagnostic of the cluster was wrong (based on a transient degraded state of the authn operator that wasn't there anymore when the attempted fix was performed), apparently some kicking the ingress controller helped (or the problem has just gone away by itself, this is libvirt/KVM after all and we have no idea about the resources the cluster has available), so I am moving the BZ to them.

Marc, if you don't have the logs anymore, I expect the routing team will need you to reproduce the problem, you may want to post must-gather right away the next time.
Comment 12 Dan Mace 2020-03-30 11:48:40 UTC 
As far as I know, this topology isn't officially supported in IPI, and given other priorities I can't promise we'll be looking into the issue any time soon. Sorry about that — just trying to help set expectations.
Comment 13 Andrew McDermott 2020-05-19 15:01:15 UTC 
Moving to 4.6.
Comment 14 Andrew McDermott 2020-06-17 11:18:59 UTC 
Adding UpcomingSprint though the priority for fixing will continue to be low.
Comment 15 Andrew McDermott 2020-07-09 12:05:44 UTC 
I’m adding UpcomingSprint, because I was occupied by fixing bugs with
higher priority/severity, developing new features with higher
priority, or developing new features to improve stability at a macro
level. I will revisit this bug next sprint.
Comment 16 Andrew McDermott 2020-07-30 10:00:45 UTC 
I’m adding UpcomingSprint, because I was occupied by fixing bugs with
higher priority/severity, developing new features with higher
priority, or developing new features to improve stability at a macro
level. I will revisit this bug next sprint.
Comment 17 mfisher 2020-08-18 19:50:26 UTC 
Moving this to target release 4.7 and deferring low priority bug; will be reevaluated during 4.7 cycle.  Please re-open if this needs to be reconsidered.
Comment 18 Red Hat Bugzilla 2023-09-15 01:29:17 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days