Bug 1824934

1. delete the cm default-ingress-cert from openshift-console
$oc -n openshift-cluster-version scale deployments/cluster-version-operator --replicas=0
$oc -n openshift-ingress-operator scale deploy/ingress-operator --replicas=0
$oc -n openshift-config-managed delete configmaps/default-ingress-cert
$ oc get cm default-ingress-cert -n openshift-console
Error from server (NotFound): configmaps "default-ingress-cert" not found

2. after a while, check the deployment console
      containers:
      - command:
        - /opt/bridge/bin/bridge
        - --public-dir=/opt/bridge/static
        - --config=/var/console-config/console-config.yaml
        - --service-ca-file=/var/service-ca/service-ca.crt
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3032707cb3bd2f6089e0ec01460bf7837bea020c445cd01b20ebb21ba8fe6983
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - sleep
              - "25"
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 150
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: console
        ports:
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 10m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/serving-cert
          name: console-serving-cert
          readOnly: true
        - mountPath: /var/oauth-config
          name: console-oauth-config
          readOnly: true
        - mountPath: /var/console-config
          name: console-config
          readOnly: true
        - mountPath: /var/service-ca
          name: service-ca
          readOnly: true
        - mountPath: /var/default-ingress-cert
          name: default-ingress-cert
          readOnly: true
        - mountPath: /etc/pki/ca-trust/extracted/pem
          name: trusted-ca-bundle
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        node-role.kubernetes.io/master: ""
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: console
      serviceAccountName: console
      terminationGracePeriodSeconds: 40
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 120
      - effect: NoExecute
        key: node.kubernetes.io/not-reachable
        operator: Exists
        tolerationSeconds: 120
      volumes:
      - name: console-serving-cert
        secret:
          defaultMode: 420
          secretName: console-serving-cert
      - name: console-oauth-config
        secret:
          defaultMode: 420
          secretName: console-oauth-config
      - configMap:
          defaultMode: 420
          name: console-config
        name: console-config
      - configMap:
          defaultMode: 420
          name: service-ca
        name: service-ca
      - configMap:
          defaultMode: 420
          name: default-ingress-cert
        name: default-ingress-cert
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: tls-ca-bundle.pem
          name: trusted-ca-bundle
        name: trusted-ca-bundle

There is no oauthEndpointCAFile refer to the service account CA (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt)
$oc logs console-operator-7fb9b776dc-4hpd4 -n openshift-console-operator
417 07:55:03.663020       1 controller.go:129] {Console Console} failed with: default-ingress-cert configmap not found
I0417 07:55:03.675821       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"41f103e4-274b-41c4-961a-86cf04d30b54", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded message changed from "RouteHealthDegraded: failed to read CA to check route health: configmaps \"default-ingress-cert\" not found" to "RouteHealthDegraded: failed to read CA to check route health: configmaps \"default-ingress-cert\" not found\nDefaultIngressCertValidationDegraded: default-ingress-cert configmap not found"
E0417 07:55:04.855146       1 status.go:74] RouteHealthDegraded FailedLoadCA failed to read CA to check route health: configmaps "default-ingress-cert" not found
E0417 07:55:06.453848       1 status.go:74] DefaultIngressCertValidationDegraded FailedGet default-ingress-cert configmap not found
E0417 07:55:06.453998       1 controller.go:129] {Console Console} failed with: default-ingress-cert configmap not found
E0417 07:55:08.453589       1 status.go:74] DefaultIngressCertValidationDegraded FailedGet default-ingress-cert configmap not found
E0417 07:55:08.453713       1 controller.go:129] {Console Console} failed with: default-ingress-cert configmap not found
E0417 07:55:10.453553       1 status.go:74] DefaultIngressCertValidationDegraded FailedGet default-ingress-cert configmap not found
E0417 07:55:10.453666       1 controller.go:129] {Console Console} failed with: default-ingress-cert configmap not found
E0417 07:55:12.453723       1 status.go:74] DefaultIngressCertValidationDegraded FailedGet default-ingress-cert configmap not found

4.5.0-0.ci-2020-04-17-071134

Operator behavior should be as follows:
- default-ingress-cert should be used if it exists, otherwise
- router-ca should be used

router-ca no longer exists in 4.5 and https://bugzilla.redhat.com/show_bug.cgi?id=1824934#c3 is unsupported. Shanan, please explain which supported scenario you are trying to mock and why you're mocking it instead of actually performing the scenario.

Nvm, discard my previous comment, I did not read the PR...

Couple of observations:
- shana reveals that the operator health check demands existence of the default-ingress-cert but is willing to deploy the pods even when that does not exist, that makes little sense
- according to the enhancement https://github.com/openshift/enhancements/blob/master/enhancements/network/default-ingress-cert-configmap.md#implementation-history, default-ingress-cert always exists, so it does not make sense to test removing it
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt should contain the very same ingress cert from the CM that you're using (among other CAs), so you may not need this fallback unless you needed to deploy console pods before the certificate becomes available

I would remove the fallback as a whole unless you have a good reason to keep it. Otherwise you should fix your health checks :)

In that case I guess we should be OK to close this issue. Maybe create a story to remove the fallback?

we did not have time to address this during this sprint

So the issue here is that the bug failed the QA, but the tested scenario is unsupported,
meaning the default-ingress-cert CM always exists, so it does not make sense to test removing it
and also scale down to zero the ingress-operator and CVO. On that we agreed with @Standa.

For that case we should either test that the changes delivered in the PR are reflected, or
close it down.

Comment #8 is there just as a reminder that we now can remove the fallback of the missing 
default-ingress-cert CM, for which we can create a story or open an BZ (which would be more 
appropriate IMO).

Also no additional backports are needed.

After chat with Ben putting on ON_QA since the scenario QA used in the FailedQA case was invalid,
please check https://bugzilla.redhat.com/show_bug.cgi?id=1824934#c7

We created a story for removing the fallback.

        - mountPath: /var/default-ingress-cert
          name: default-ingress-cert
console use the /var/default-ingress-cert defauly as expected.
Verify this bug
4.5.0-0.nightly-2020-05-20-183547

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days