Related: https://chat.google.com/room/AAAAWskU424/gYnzkoh4Znk

@Tomas Coufal,

In 2.2.3, the log indicates custom ingress certificate SAN doesn't contain localhost. While localhost is necessary referring to doc https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.2/html/security/security#replacing-the-management-ingress-certificates .

As you mentioned "The same certificate worked just fine in 2.2.2 and 2.2.1", it's strange to me. 
I just deploy a 2.2.2 cluster and try to reproduce it. I create a custom certificate with the below csr.cnf
```
# cat csr.cnf
[ req ]               # Main settings
default_bits = 2048       # Default key size in bits.
prompt = no               # Disables prompting for certificate values so the configuration file values are used.
default_md = sha256       # Specifies the digest algorithm.
req_extensions = req_ext  # Specifies the configuration file section that includes any extensions.
distinguished_name = dn   # Specifies the section that includes the distinguished name information.

[ dn ]               # Distinguished name settings
C = US                    # Country
ST = North Carolina             # State or province
L = Raleigh                # Locality
O = Red Hat Open Shift     # Organization
OU = Red Hat Advanced Container Management        # Organizational unit
CN = management-ingress  # Common name.

[ req_ext ]          # Extensions
subjectAltName = @alt_names # Subject alternative names

[ alt_names ]        # Subject alternative names
DNS.1 = management-ingress
DNS.2 = multicloud-console.apps.obs-china-aws-4616-bjrps.dev05.red-chesterfield.com
DNS.3 = 127.0.0.1

```
Then meet the same log as your 2.3 cluster.
```
# oco logs management-ingress-14761-fd4dccf78-dfwxs -c oauth-proxy
...
2021/06/18 02:25:33 reverseproxy.go:437: http: proxy error: x509: certificate is valid for management-ingress, multicloud-console.apps.obs-china-aws-4616-bjrps.dev05.red-chesterfield.com, 127.0.0.1, not localhost

```

Could you share your 2.2.2 management-ingress deploy info to help me understand why your 2.2.2 can work without localhost in cert?
```
# oc get deploy -n open-cluster-management | grep ingress
management-ingress-14761                         2/2     2            2           42m
#  oc get deploy -n open-cluster-management management-ingress-14761 -oyaml > management-ingress.yaml
```

Below is my deployment if you want to compare
```
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    meta.helm.sh/release-name: management-ingress-14761
    meta.helm.sh/release-namespace: open-cluster-management
  creationTimestamp: "2021-06-18T01:58:20Z"
  generation: 4
  labels:
    app: management-ingress-14761
    app.kubernetes.io/instance: management-ingress-14761
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: management-ingress-14761
    certmanager.k8s.io/time-restarted: 2021-6-18.0158
    chart: management-ingress-2.2.2
    component: management-ingress-14761
    helm.sh/chart: management-ingress
    heritage: Helm
    installer.name: multiclusterhub
    installer.namespace: open-cluster-management
    release: management-ingress-14761
  name: management-ingress-14761
  namespace: open-cluster-management
  ownerReferences:
  - apiVersion: apps.open-cluster-management.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: HelmRelease
    name: management-ingress-14761
    uid: 1ce56bec-1d7e-4e57-92d8-29ffe73863d3
  resourceVersion: "69599"
  selfLink: /apis/apps/v1/namespaces/open-cluster-management/deployments/management-ingress-14761
  uid: 80f65aa6-5679-4ce0-9059-5bc23282fe12
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: management-ingress-14761
      chart: management-ingress
      component: management-ingress-14761
      heritage: Helm
      k8s-app: management-ingress-14761
      release: management-ingress-14761
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        productID: management-ingress_2.2.2_00000
        productName: management-ingress
        productVersion: 2.2.2
      creationTimestamp: null
      labels:
        app: management-ingress-14761
        app.kubernetes.io/instance: management-ingress-14761
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: management-ingress-14761
        certmanager.k8s.io/time-restarted: 2021-6-18.0158
        chart: management-ingress
        component: management-ingress-14761
        helm.sh/chart: management-ingress
        heritage: Helm
        k8s-app: management-ingress-14761
        ocm-antiaffinity-selector: managementingress
        release: management-ingress-14761
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: ocm-antiaffinity-selector
                  operator: In
                  values:
                  - managementingress
              topologyKey: topology.kubernetes.io/zone
            weight: 70
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: ocm-antiaffinity-selector
                  operator: In
                  values:
                  - managementingress
              topologyKey: kubernetes.io/hostname
            weight: 35
      containers:
      - args:
        - --provider=openshift
        - --upstream=https://localhost:8443
        - --upstream-ca=/etc/tls/ca/tls.crt
        - --https-address=:9443
        - --client-id=multicloudingress
        - --client-secret=multicloudingresssecret
        - --pass-user-bearer-token=true
        - --pass-access-token=true
        - --scope=user:full
        - '-openshift-delegate-urls={"/": {"resource": "projects", "verb": "list"}}'
        - --skip-provider-button=true
        - --cookie-secure=true
        - --cookie-expire=12h0m0s
        - --cookie-refresh=8h0m0s
        - --tls-cert=/etc/tls/private/tls.crt
        - --tls-key=/etc/tls/private/tls.key
        - --cookie-secret=AAECAwQFBgcICQoLDA0OFw==
        - --openshift-ca=/etc/pki/tls/cert.pem
        - --openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        image: quay.io/open-cluster-management/origin-oauth-proxy@sha256:ae1b3ad6a9e76271de5c6d16abe5586abac043da032d686d1fc78f43ed7f3606
        imagePullPolicy: Always
        name: oauth-proxy
        ports:
        - containerPort: 9443
          name: public
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /oauth/healthz
            port: 9443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/tls/private
          name: tls-secret
        - mountPath: /etc/tls/ca
          name: ca-tls-secret
      - command:
        - /management-ingress
        - --default-ssl-certificate=$(POD_NAMESPACE)/byo-ingress-tls-secret
        - --configmap=$(POD_NAMESPACE)/management-ingress-14761
        - --http-port=8080
        - --https-port=8443
        env:
        - name: ENABLE_IMPERSONATION
          value: "false"
        - name: APISERVER_SECURE_PORT
          value: "8001"
        - name: CLUSTER_DOMAIN
          value: cluster.local
        - name: HOST_HEADERS_CHECK_ENABLED
          value: "false"
        - name: ALLOWED_HOST_HEADERS
          value: 127.0.0.1 localhost management-ingress-14761 management-ingress multicloud-console.apps.obs-china-aws-4616-bjrps.dev05.red-chesterfield.com
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: FIPS_ENABLED
          value: "false"
        image: quay.io/open-cluster-management/management-ingress@sha256:18ebc2d28236fa8da93bae1dab8a532e65043cd4b3c9f45ebe0757d90e7abe55
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: management-ingress-14761
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 8443
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: true
          runAsNonRoot: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/secrets/tls
          name: tls-secret
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: multiclusterhub-operator-pull-secret
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: management-ingress-14761-sa
      serviceAccountName: management-ingress-14761-sa
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: dedicated
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        operator: Exists
      volumes:
      - name: tls-secret
        secret:
          defaultMode: 420
          secretName: byo-ingress-tls-secret
      - name: ca-tls-secret
        secret:
          defaultMode: 420
          secretName: byo-ca-cert
```

I think we misunderstood the reason why we would replace the ingress cert in ACM. Instead we should have replaced the OCP ingress cert.

No need to reopen this.