Bug 1807656

Summary: OLM's TestRHOperators fails if an Operator is not installed
Product: OpenShift Container Platform Reporter: Alexander Greene <agreene>
Component: OLMAssignee: Bowen Song <bsong>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED WONTFIX Docs Contact:
Severity: unspecified    
Priority: unspecified    
Version: 4.3.z   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-03 22:15:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Greene 2020-02-26 20:44:26 UTC
Description of problem:
OLM has a periodic test that installs all Red Hat Operators [1 and 2]
This test has been failing since inception because a RH Operator is failing Schema Validation as shown in [3]:
```
error waiting for CSV to become ready, operator businessautomation-operator install plan is in phase: Failed, InstallComponentFailed, error creating csv businessautomation-operator.1.3.0: ClusterServiceVersion.operators.coreos.com "businessautomation-operator.1.3.0" is invalid: spec.customresourcedefinitions.owned.specDescriptors.value: Invalid value: "": spec.customresourcedefinitions.owned.specDescriptors.value in body must be of type byte: "": timed out waiting for the condition
```

This test will eventually be updated to perform metric tests against each of these operators. If an operator fails to install, it does not overload metrics and should not fail this test.


How reproducible:
Every time the test runs.

Steps to Reproduce:
1. Visit [3]

Actual results:
Test fails

Expected results:
Test continues to run despite failing to install an operator.

Additional info:

[1] https://github.com/operator-framework/operator-lifecycle-manager/blob/master/test/rh-operators/metric_test.go
[2] https://github.com/openshift/release/blob/master/ci-operator/jobs/operator-framework/operator-lifecycle-manager/operator-framework-operator-lifecycle-manager-master-periodics.yaml
[3] https://prow.svc.ci.openshift.org/job-history/origin-ci-test/logs/periodic-ci-operator-framework-operator-lifecycle-managment-master-rhoperator-metric-e2e-aws-olm-master-daily