Bug 1907290

Created attachment 1738865 [details]
OLM operator log

Description of problem:
HCO openshift-ci failes on AWS, After upgrading CNV, while trying to send a request to hco-webhook, with the following error:

> Error from server (InternalError): Internal error occurred: failed calling webhook "mutate-ns-hco.kubevirt.io": Post "https://hco-webhook-service.kubevirt-hyperconverged.svc:4343/mutate-ns-hco-kubevirt-io?timeout=30s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "Red Hat, Inc.")

For example: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/991/pull-ci-kubevirt-hyperconverged-cluster-operator-master-hco-e2e-upgrade-prev-azure/1336649438625533952
 

Here is where the test installs the "old" version:
https://github.com/kubevirt/hyperconverged-cluster-operator/blob/d1efad33eb3e6624e4ee337f593218239e86ea48/hack/upgrade-test.sh#L165-L177

And here is where the test updates the CSV version for upgrade:
https://github.com/kubevirt/hyperconverged-cluster-operator/blob/d1efad33eb3e6624e4ee337f593218239e86ea48/hack/upgrade-test.sh#L215


It was reporoduced in different scenario: using this index image: 

> quay.io/nunnatsa/hyperconverged-cluster-index:1.3.0 

1. We installed the 1.3.0 channel, then had to fix its CSV manually (another issue) to remove an annotations.description field from one of the templates, then it completed the installation.
2. uninstall 1.3.0 (originally, in order to start the upgrade scenario from 1.2.0)
3. install 1.2.0
4. trying to deploy the HyperConverged CR:
> cat <<EOF | oc} create -n kubevirt-hyperconverged -f -
> apiVersion: hco.kubevirt.io/v1beta1
> kind: HyperConverged
> metadata:
>  name: kubevirt-hyperconverged
> spec:
>   infra: {}
>   workloads: {}
> EOF

At this point, we got this error:
> Error from server (InternalError): error when creating "deploy/hco.cr.yaml": Internal error occurred: failed calling webhook "validate-hco.kubevirt.io": Post "https://hco-webhook-service.kubevirt-hyperconverged.svc:4343/validate-hco-kubevirt-io-v1beta1-hyperconverged?timeout=30s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "Red Hat, Inc.")


OLM operator log is attached.