Bug 1811417

Summary: Failed to install cnv-2.4 on top of ocp 4.4 (hco operator in crashLoopBackOff state)
Product: Container Native Virtualization (CNV) Reporter: Tareq Alayan <talayan>
Component: InstallationAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED ERRATA QA Contact: Satyajit Bulage <sbulage>
Severity: high Docs Contact:
Priority: high    
Version: 2.4.0CC: awels, cnv-qe-bugs, ncredi, pelauter, sbulage, stirabos
Target Milestone: ---Keywords: Regression
Target Release: 2.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: hostpath-provisioner-operator-container-v2.4.0-19 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-28 19:09:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tareq Alayan 2020-03-08 14:22:06 UTC
Description of problem:
I tried to install cnv-2.4 on top of ocp-4.4 by running this script[1]

I saw that the pod
hco-operator-7b55489755-n5l5p                     0/1     CrashLoopBackOff   7          16m

I ran oc describe pod hco-operator-7b55489755-n5l5p -nopenshift-cnv
Normal   Pulled     16m                  kubelet, talayan-hpp2-8k6lw-worker-zlgzr  Successfully pulled image "registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hyperconverged-cluster-operator@sha256:f9fa03a03ce7507a0fe17d2ce5ade1e548f7bfebe797c5ef43a3956415c29972"
  Warning  Unhealthy  16m                  kubelet, talayan-hpp2-8k6lw-worker-zlgzr  Readiness probe errored: rpc error: code = Unknown desc = container is not created or running

full describe output http://pastebin.test.redhat.com/842336

[1] http://pastebin.test.redhat.com/842339

Comment 2 Simone Tiraboschi 2020-03-09 17:18:07 UTC
Not reproducible on my side with registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hyperconverged-cluster-operator@sha256:e2170e4eb356b152f984670723ebd3e85cb3c75d7ea7c1ddefd7ebed1feb3d09 ( container-native-virtualization-hyperconverged-cluster-operator:v2.4.0-24 ).



Tarek, can you please attach also the output of
  oc logs -n openshift-cnv  $(oc get pods -n openshift-cnv -l name=hyperconverged-cluster-operator -o name)
?

Comment 9 Tareq Alayan 2020-03-16 11:49:23 UTC
I tried to deploy again:
NAME                                                  READY   STATUS    RESTARTS   AGE
bridge-marker-6cnrj                                   1/1     Running   0          14m
bridge-marker-6l54x                                   1/1     Running   0          14m
bridge-marker-6vfg9                                   1/1     Running   0          14m
bridge-marker-n5nqz                                   1/1     Running   0          14m
bridge-marker-qwxrp                                   1/1     Running   0          14m
bridge-marker-zvdvs                                   1/1     Running   0          14m
cdi-apiserver-7476846b98-jq69g                        1/1     Running   0          14m
cdi-deployment-c4f89465-f6mw6                         1/1     Running   0          14m
cdi-operator-7bdcc7645-mxkr5                          1/1     Running   0          15m
cdi-uploadproxy-bb7bdcdd4-pttwn                       1/1     Running   0          14m
cluster-network-addons-operator-5f68d8d494-wgt5r      1/1     Running   0          15m
hco-operator-67cd964587-r4mnh                         1/1     Running   0          15m
hostpath-provisioner-operator-775d7fccc4-j5hfl        1/1     Running   0          15m
kube-cni-linux-bridge-plugin-6v7pm                    1/1     Running   0          14m
kube-cni-linux-bridge-plugin-kv2xv                    1/1     Running   0          14m
kube-cni-linux-bridge-plugin-l876t                    1/1     Running   0          14m
kube-cni-linux-bridge-plugin-mx7dk                    1/1     Running   0          14m
kube-cni-linux-bridge-plugin-swx5v                    1/1     Running   1          14m
kube-cni-linux-bridge-plugin-v7467                    1/1     Running   0          14m
kubemacpool-mac-controller-manager-65f6cc86b4-drjnv   1/1     Running   0          14m
kubemacpool-mac-controller-manager-65f6cc86b4-t7r2k   1/1     Running   0          14m
kubevirt-node-labeller-96tds                          1/1     Running   0          13m
kubevirt-node-labeller-jvz4j                          1/1     Running   0          13m
kubevirt-node-labeller-rj9cr                          1/1     Running   0          13m
kubevirt-ssp-operator-85d85d77cb-m9klx                1/1     Running   0          15m
nmstate-handler-fw9tx                                 1/1     Running   0          14m
nmstate-handler-jdn94                                 1/1     Running   0          14m
nmstate-handler-q9kg5                                 1/1     Running   0          14m
nmstate-handler-worker-2sqtm                          1/1     Running   0          14m
nmstate-handler-worker-bsdjg                          1/1     Running   0          14m
nmstate-handler-worker-qgcbr                          1/1     Running   0          14m
node-maintenance-operator-7696458f68-ng982            1/1     Running   0          15m
ovs-cni-amd64-454nr                                   2/2     Running   0          14m
ovs-cni-amd64-h65dx                                   2/2     Running   0          14m
ovs-cni-amd64-rt9ns                                   2/2     Running   0          14m
ovs-cni-amd64-skqvp                                   2/2     Running   0          14m
ovs-cni-amd64-t6cw5                                   2/2     Running   0          14m
ovs-cni-amd64-z4tjm                                   2/2     Running   0          14m
virt-api-7b9f9fd6d5-5rkvt                             1/1     Running   0          14m
virt-api-7b9f9fd6d5-z4cb7                             1/1     Running   0          14m
virt-controller-75f55cb978-jwqd9                      1/1     Running   0          13m
virt-controller-75f55cb978-s2xzx                      1/1     Running   0          13m
virt-handler-7smxt                                    1/1     Running   0          13m
virt-handler-k7dqp                                    1/1     Running   0          13m
virt-handler-l7fjf                                    1/1     Running   0          13m
virt-operator-66dd8854bf-2rcjc                        1/1     Running   0          15m
virt-operator-66dd8854bf-zhw4k                        1/1     Running   0          15m
virt-template-validator-55b9d4fc6b-2lhpv              1/1     Running   0          13m
virt-template-validator-55b9d4fc6b-2vtzt              1/1     Running   0          13m
[cloud-user@ocp-psi-executor talayan]$ oc get csv -nopenshift-cnv 
NAME                                      DISPLAY                           VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v2.4.0   Container-native virtualization   2.4.0                Succeeded

Comment 10 Nelly Credi 2020-03-23 07:52:20 UTC
Please add 'fixed in version'

Comment 12 Simone Tiraboschi 2020-03-24 09:31:46 UTC
(In reply to Satyajit Bulage from comment #11)
> Readiness probe failed: stat: cannot stat '/tmp/operator-sdk-ready': No such file or directory

This simply means that one of dependent operators is not ready.
You have to check conditions on the kubevirt-hyperconverged hyperconverged resource to identify the real issue.
Can you please attach the output of
 oc get -n openshift-cnv hyperconverged kubevirt-hyperconverged -o yaml
?

Comment 14 Satyajit Bulage 2020-03-31 09:31:24 UTC
As per comment 11, I am not able to see crashLoopBackOff error.

Verifying BZ.

Comment 17 errata-xmlrpc 2020-07-28 19:09:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3194