Bug 1876919

I reproduced this on a stage OSD cluster by

1. installing 4.5.7
2. verified oauth and console worked
3. added github identity provider in OCM
4. verified console is not working

Note on OSD in stage we use LDAP for authn, so initial verification is with one valid identity provider.

Since this didn't work I tried to revert by taking the new github identity provider out of oauth and adding it back in.  After doing this authn works.  I was not able to fix the production cluster with this issue though.  It shows authentication degraded so may be exhibiting an additional problem.

Tagging with 4.7 while we investigate.

Trying to see if I can figure out something concrete from the OSD perspective to reproduce.  Over lunch I created a new 4.5.7 cluster and let it bake a while.  Then I added a github identity provider and let that roll out, coming back maybe 30 min later.  Everything was working fine.

My next test is to add a second identity provider right after the cluster is installed by hive.  I will not wait for OSD things such as certificates to land completely.  This is how customers with automation (including the one that is the cause of this alert) are provisioning, via api and setting things up as soon as permitted.

The following errors logged repeatedly by the auth operator (as per #c12) is log spam:

E0909 19:49:58.876033       1 base_controller.go:180] "ConfigObserver" controller failed to sync "key", err: .servingInfo.namedCertificates accessor error: <nil> is of the type <nil>, expected []interface{}

This was fixed for 4.6 in https://github.com/openshift/cluster-authentication-operator/pull/288, and I'll make sure this gets backported to 4.5.

The other conditions reported by the auth operator indicate a problem with reading the openshift-config-managed/router-certs secret. The error messages are admittedly obscure, and the auth team is working on ensuring we report more informative conditions: 

    message: |-
      ConfigObservationDegraded: .servingInfo.namedCertificates accessor error: <nil> is of the type <nil>, expected []interface{}
      RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.fastt02.i8v0.p1.openshiftapps.com] -n openshift-authentication: not found
      RouteHealthDegraded: failed to GET route: dial tcp: lookup oauth-openshift.apps.fastt02.i8v0.p1.openshiftapps.com on 172.30.0.10:53: no such host
    reason: ConfigObservation_Error::RouteHealth_FailedGet::RouterCerts_MissingRouterCertsPEM

I looked at the router-certs secret on a cluster provided by nmalik and the router-certs secret was as follows:

kind: Secret
metadata:
  creationTimestamp: "2020-09-09T17:03:37Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:type: {}
    manager: ingress-operator
    operation: Update
    time: "2020-09-09T17:24:14Z"
  name: router-certs
  namespace: openshift-config-managed
  resourceVersion: "37523"
  selfLink: /api/v1/namespaces/openshift-config-managed/secrets/router-certs
  uid: 87ac619b-9de7-4c26-99b2-02b2e6e0990d
type: Opaque

There were no certs present in the secret. This is an unusual condition, and one best investigated by the ingress team.

Adding link to backport of fix for auth operator log spam.

*** This bug has been marked as a duplicate of bug 1878148 ***