Description of problem: CNV v4.11.0.rhel9-285 installation fails because there's a problem with the HCO. NAME READY STATUS RESTARTS AGE hco-operator-77bdb4b8cd-fq22n 0/1 CrashLoopBackOff 10 (2m52s ago) 30m hco-webhook-85fcc6b654-h2b4j 0/1 CrashLoopBackOff 10 (3m10s ago) 30m From hco-webhook pod: Warning Unhealthy 28m kubelet Readiness probe failed: Get "http://10.129.2.162:6060/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 28m kubelet Readiness probe failed: Get "http://10.129.2.162:6060/readyz": read tcp 10.129.2.1:47802->10.129.2.162:6060: read: connection reset by peer Warning ProbeError 28m kubelet Readiness probe error: Get "http://10.129.2.162:6060/readyz": read tcp 10.129.2.1:47802->10.129.2.162:6060: read: connection reset by peer body: Warning ProbeError 27m kubelet Readiness probe error: Get "http://10.129.2.162:6060/readyz": read tcp 10.129.2.1:48118->10.129.2.162:6060: read: connection reset by peer Logs from hco-operator pod: {"level":"error","ts":1649920076.2278628,"logger":"hyperconverged-operator-cmd","msg":"Cannot detect cluster type","error":"apiservers.config.openshift.io \"cluster\" is forbidden: User \"system:serviceaccount:openshift-cnv:hyperconverged-cluster-operator\" cannot get resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/cmd/cmdcommon.HcCmdHelper.ExitOnError\n\t/remote-source/app/cmd/cmdcommon/cmdcommon.go:108\nmain.main\n\t/remote-source/app/cmd/hyperconverged-cluster-operator/main.go:120\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:255"} Version-Release number of selected component (if applicable): CNV v4.11.0.rhel9-285 IIB 213855 How reproducible: 100 Steps to Reproduce: 1. Install the above mentioned CNV version. 2. 3. Actual results: Installation fails.
Correction: TTO was not added yet to hco-csv-generator script for 4.11-rhel9.
Verified with v4.11.0.rhel9-376: registry-proxy.engineering.redhat.com/rh-osbs/iib:235198 [cnv-qe-jenkins@c01-dbn-411r9-hcnfv-executor ~]$ kubectl get pods -n openshift-cnv NAME READY STATUS RESTARTS AGE bridge-marker-7g6qj 1/1 Running 0 8m37s bridge-marker-drlms 1/1 Running 0 8m37s bridge-marker-k62qr 1/1 Running 0 8m37s bridge-marker-s4p5m 1/1 Running 0 8m37s bridge-marker-s8w26 1/1 Running 0 8m37s bridge-marker-smvhs 1/1 Running 0 8m37s cdi-apiserver-6bb98d5f87-dtxs5 1/1 Running 0 8m36s cdi-deployment-676559bd59-n9sqt 1/1 Running 0 8m37s cdi-operator-74c9669d6f-wb8v4 1/1 Running 0 9m58s cdi-uploadproxy-6878bdbb9c-bb7kk 1/1 Running 0 8m36s cluster-network-addons-operator-55c9fd64b8-psx7n 2/2 Running 0 10m hco-operator-64f4d7bfb6-pvq9h 1/1 Running 0 10m hco-webhook-f4484479f-chwjh 1/1 Running 0 10m hostpath-provisioner-csi-dzwp4 4/4 Running 0 6m59s hostpath-provisioner-csi-tn8b7 4/4 Running 0 6m59s hostpath-provisioner-csi-zqqnv 4/4 Running 0 6m59s hostpath-provisioner-operator-766f9c8874-x4qf5 1/1 Running 0 9m57s hpp-pool-hpp-csi-pvc-block-c01-dbn-411r9-hcnfv-worker-0-5slw4zp 1/1 Running 0 6m59s hpp-pool-hpp-csi-pvc-block-c01-dbn-411r9-hcnfv-worker-0-h5tzttw 1/1 Running 0 6m59s hpp-pool-hpp-csi-pvc-block-c01-dbn-411r9-hcnfv-worker-0-mmkr877 1/1 Running 0 6m59s hyperconverged-cluster-cli-download-548474b55b-h8fwx 1/1 Running 0 10m kube-cni-linux-bridge-plugin-htnbr 1/1 Running 0 8m37s kube-cni-linux-bridge-plugin-l88sq 1/1 Running 0 8m37s kube-cni-linux-bridge-plugin-lmjth 1/1 Running 0 8m37s kube-cni-linux-bridge-plugin-pg9st 1/1 Running 0 8m37s kube-cni-linux-bridge-plugin-psdsr 1/1 Running 0 8m37s kube-cni-linux-bridge-plugin-x9vcm 1/1 Running 0 8m37s kubemacpool-cert-manager-75695d997-ggd2m 1/1 Running 0 8m36s kubemacpool-mac-controller-manager-7474c4ccf4-nk7ns 2/2 Running 0 8m36s kubevirt-plugin-5789769948-bq64j 1/1 Running 0 8m15s node-maintenance-operator-controller-manager-8645947467-gjdbv 1/1 Running 0 10m ssp-operator-bc648cc64-t267d 1/1 Running 0 10m tekton-tasks-operator-67584f8455-drpkp 1/1 Running 0 10m virt-api-cb4f88967-bx9lk 1/1 Running 0 8m16s virt-api-cb4f88967-x5tg9 1/1 Running 0 8m16s virt-controller-7b446d68dd-ks79v 1/1 Running 0 7m50s virt-controller-7b446d68dd-zh9xm 1/1 Running 0 7m50s virt-handler-j8bxv 1/1 Running 0 7m50s virt-handler-llhlp 1/1 Running 0 7m50s virt-handler-s5t7s 1/1 Running 0 7m50s virt-operator-b74955b99-rw79h 1/1 Running 0 9m39s virt-operator-b74955b99-tscw2 1/1 Running 0 9m39s virt-template-validator-7887d487bd-h7dpx 1/1 Running 0 8m21s virt-template-validator-7887d487bd-rj2r4 1/1 Running 0 8m21s [cnv-qe-jenkins@c01-dbn-411r9-hcnfv-executor ~]$ kubectl get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE jaeger-operator.v1.30.2 Red Hat OpenShift distributed tracing platform 1.30.2 Succeeded kiali-operator.v1.36.9 Kiali Operator 1.36.9 kiali-operator.v1.36.8 Succeeded kubevirt-hyperconverged-operator.v4.11.0 OpenShift Virtualization 4.11.0 kubevirt-hyperconverged-operator.v4.10.0 Succeeded servicemeshoperator.v2.1.2.1 Red Hat OpenShift Service Mesh 2.1.2-1 servicemeshoperator.v2.1.2 Succeeded [cnv-qe-jenkins@c01-dbn-411r9-hcnfv-executor ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.11.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6526