Bug 1881713

Summary: `TestMarketplace/no-setup-test-group/operatorhub-test-suite/disable-all-enable-one` test is failing on OKD configuration
Product: OpenShift Container Platform Reporter: Vadim Rutkovsky <vrutkovs>
Component: OLMAssignee: Ankita Thomas <ankithom>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: high CC: anbhatta, ankithom
Version: 4.6   
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:44:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vadim Rutkovsky 2020-09-22 22:56:48 UTC 
Description of problem:

See https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/11529/rehearse-11529-pull-ci-operator-framework-operator-marketplace-master-okd-e2e-aws-operator/1308505582092685312
```
             --- FAIL: TestMarketplace/no-setup-test-group/operatorhub-test-suite/disable-all-enable-one (5.29s)
            	operatorhubtests.go:97: 
            			Error Trace:	operatorhubtests.go:97
            			Error:      	Received unexpected error:
            			            	default CatalogSource still present on the cluster
            			Test:       	TestMarketplace/no-setup-test-group/operatorhub-test-suite/disable-all-enable-one
            			Messages:   	Default CatalogSource has not been disabled 
```

This is probably happening due to a different default config used in OKD - https://github.com/openshift/installer/blob/fcos/data/data/manifests/openshift/community-operators.yaml
Comment 1 Anik 2020-09-24 14:00:56 UTC 
@Vadim, the test is supposed to override whatever config the cluster has before the test start: https://github.com/operator-framework/operator-marketplace/blob/2d6d7189243754f3ab8da86b3e7aa39c6fcdbbab/test/testsuites/operatorhubtests.go#L82

As long as all the defaults are present on the cluster, even if some are disabled, this test shouldn't be affected. 

How often is this happening?
Comment 2 Vadim Rutkovsky 2020-09-24 14:05:16 UTC 
(In reply to Anik from comment #1)
> How often is this happening?

I've seen it once so far, as OKD tests are not being run regularly until https://github.com/operator-framework/operator-marketplace/pull/332 is merged.
Comment 3 Vadim Rutkovsky 2020-09-24 14:05:45 UTC 
Wrong PR number - https://github.com/openshift/release/pull/11529 would add optional okd test to marketplace PRs
Comment 4 Anik 2020-09-24 15:07:44 UTC 
This test is likely to flip flop between passing and failing because of https://github.com/operator-framework/operator-marketplace/blob/2d6d7189243754f3ab8da86b3e7aa39c6fcdbbab/test/testsuites/operatorhubtests.go#L93

https://github.com/operator-framework/operator-marketplace/blob/2d6d7189243754f3ab8da86b3e7aa39c6fcdbbab/test/testsuites/operatorhubtests.go#L449 this does not poll for the correct status of the cluster. 

https://github.com/operator-framework/operator-marketplace/blob/2d6d7189243754f3ab8da86b3e7aa39c6fcdbbab/test/testsuites/operatorhubtests.go#L407 should be called instead to ensure higher success rate. 

Setting the priority for this as low for now.
Comment 5 Anik 2020-09-24 16:45:42 UTC 
Looks like this might be blocking a 4.6 blocker bug https://bugzilla.redhat.com/show_bug.cgi?id=1874751.

Changing priority and target release so that the fix for this can unblock 1874751
Comment 7 Jian Zhang 2020-09-27 06:48:11 UTC 
It works well, LGTM, verify it.
Comment 10 errata-xmlrpc 2020-10-27 16:44:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196