Bug 1876908

Summary:	KubeVirtMetricsAggregation is missing the "Available" condition
Product:	Container Native Virtualization (CNV)	Reporter:	Oren Cohen <ocohen>
Component:	SSP	Assignee:	Omer Yahud <oyahud>
Status:	CLOSED ERRATA	QA Contact:	Denis Ollier <dollierp>
Severity:	medium	Docs Contact:
Priority:	urgent
Version:	2.5.0	CC:	cnv-qe-bugs, dollierp, fdeutsch, msluiter, oyahud, stirabos
Target Milestone:	---	Keywords:	AutomationBlocker, TestBlocker
Target Release:	2.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	kubevirt-ssp-operator-container-v2.5.0-35, hco-bundle-registry-container-v2.5.0-163	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-17 13:24:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1868229

Description Oren Cohen 2020-09-08 13:30:38 UTC

Description of problem:
When deploying Openshift Virtualization 2.5.0 using the following index image:
registry-proxy.engineering.redhat.com/rh-osbs/iib:7848
SSP's KubeVirtMetricsAggregation CR is not exposing the "Available" condition.
It has only the "Running" condition set to "true" indefinetly.
This causes HCO not to become ready, and we are hitting https://bugzilla.redhat.com/show_bug.cgi?id=1868712 which affects both hco and nmo pods.

Version-Release number of selected component (if applicable):
SSP Operator image:
registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-kubevirt-ssp-operator@sha256:8aa81bf1beab8bc7bb689bef2e68e1db594d2dccaa960baed8088c9365e74db7

https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/kubevirt-ssp-operator/images/v2.5.0-33

OCP: 4.6.0-fc.3

HCO image:
registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hyperconverged-cluster-operator@sha256:e93355c3e8d98e38630342116aca46186c8873dda08e2449801fb040880f8b82

https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/hyperconverged-cluster-operator/images/v2.5.0-29

index-image:
registry-proxy.engineering.redhat.com/rh-osbs/iib:7848 

How reproducible:
100%

Steps to Reproduce:
1. Deploy CNV 2.5.0 using the above index-image as a catalog source.
2.
3.

Actual results:
KubeVirtMetricsAggregation doesn't have an "Available" condition

Expected results:
KubeVirtMetricsAggregation should eventually expose "Available=true" condition.

Additional info:
HCO CR conditions:
  - lastHeartbeatTime: "2020-09-08T13:28:09Z"
    lastTransitionTime: "2020-09-08T13:08:06Z"
    message: Reconcile completed successfully
    reason: ReconcileCompleted
    status: "True"
    type: ReconcileComplete
  - lastHeartbeatTime: "2020-09-08T13:28:09Z"
    lastTransitionTime: "2020-09-08T13:08:04Z"
    message: missing "Available" condition
    reason: KubeVirtMetricsAggregationNotAvailable
    status: "False"
    type: Available
  - lastHeartbeatTime: "2020-09-08T13:28:09Z"
    lastTransitionTime: "2020-09-08T13:10:01Z"
    message: Reconcile completed successfully
    reason: ReconcileCompleted
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2020-09-08T13:28:09Z"
    lastTransitionTime: "2020-09-08T13:08:04Z"
    message: Reconcile completed successfully
    reason: ReconcileCompleted
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2020-09-08T13:28:09Z"
    lastTransitionTime: "2020-09-08T13:10:01Z"
    message: Reconcile completed successfully
    reason: ReconcileCompleted
    status: "True"
    type: Upgradeable

$ oc get KubeVirtMetricsAggregation metrics-aggregation-kubevirt-hyperconverged -o yaml
[...]
status:
  conditions:
  - ansibleResult:
      changed: 2
      completion: 2020-09-08T13:08:48.894215
      failures: 0
      ok: 4
      skipped: 0
    lastTransitionTime: "2020-09-08T13:08:40Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running
  operatorVersion: v2.5.0
  targetVersion: v2.5.0


SSP pod logs (complete):
https://drive.google.com/file/d/1nosQgKzf2wqXKB8RWLbz2168R7SPGa3k/view?usp=sharing

Comment 1 Denis Ollier 2020-09-08 17:55:21 UTC

A recurring error in the logs is:

> fatal: [localhost]: FAILED! => {"msg": "template error while templating string: no filter named 'k8s_inject_ownership'. String: {{ objects | k8s_inject_ownership(cr_info) }}"}

The k8s_inject_ownership filter is part of kubevirt-ssp-operator since upstream PR#202 (https://github.com/kubevirt/kubevirt-ssp-operator/pull/202).

However it is missing from the downstream kubevirt-ssp-operator container image (registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-kubevirt-ssp-operator:v2.5.0-33).

We need to backport upstream changes in the Dockerfile from PR#202 (https://github.com/kubevirt/kubevirt-ssp-operator/pull/202/files#diff-4bb306eddc70a07a963059d2c51ad097) to our downstream Dockerfile (http://pkgs.devel.redhat.com/cgit/containers/kubevirt-ssp-operator/tree/Dockerfile.in?h=cnv-2.5-rhel-7) to also copy the filter_plugins/ (https://github.com/kubevirt/kubevirt-ssp-operator/tree/master/filter_plugins) directory.

Comment 3 Omer Yahud 2020-09-09 09:35:02 UTC

Hi, I am updating the DS container right now with the proper upstream commit and Dockerfile

Comment 4 Omer Yahud 2020-09-09 11:05:43 UTC

Fix was submitted, waiting for automation to build the new container.
Will update the bug when the container is built.

Comment 5 Omer Yahud 2020-09-09 13:25:08 UTC

Waiting for an hco-bundle build, ssp-build is ready

Comment 6 Denis Ollier 2020-09-10 08:37:00 UTC

Verified with registry-proxy.engineering.redhat.com/rh-osbs/iib:8258.

Comment 9 errata-xmlrpc 2020-11-17 13:24:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5127