Bug 1896160 - CI: Some cluster operators are not ready: marketplace (missing: Degraded)
Summary: CI: Some cluster operators are not ready: marketplace (missing: Degraded)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Alexander Greene
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1896204
TreeView+ depends on / blocked
 
Reported: 2020-11-09 20:44 UTC by W. Trevor King
Modified: 2021-02-24 15:31 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The Marketplace Operator did not always report the degraded ClusterStatusConditionType when reporting its status to CVO. Consequence: It was possible for the marketplace to have no Degraded ClusterStatusConditionType, a field expected by cluster admins. Fix: Marketplace will always reports its degraded state when updating the ClusterOperator status. Result: The degraded condition is present whenever the Marketplace Operator updates its ClusterOperator status.
Clone Of:
Environment:
Last Closed: 2021-02-24 15:31:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-marketplace pull 364 0 None closed Bug 1896160: Marketplace should report when it is degraded 2021-02-11 18:00:44 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:31:58 UTC

Description W. Trevor King 2020-11-09 20:44:58 UTC
Description of problem:

[sig-arch][Early] Managed cluster should start all core operators [Suite:openshift/conformance/parallel] expand_more

is occasionally failing in CI:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=24h&type=junit&search=Some%20cluster%20operators%20are%20not%20ready:%20marketplace.*missing:%20Degraded' | grep 'failures match' | sort
pull-ci-openshift-cloud-credential-operator-master-e2e-aws - 3 runs, 100% failed, 33% of failures match
pull-ci-openshift-prometheus-master-e2e-aws - 5 runs, 100% failed, 20% of failures match
release-openshift-ocp-installer-e2e-aws-4.7 - 4 runs, 50% failed, 50% of failures match
release-openshift-ocp-installer-e2e-gcp-rt-4.7 - 2 runs, 50% failed, 100% of failures match
release-openshift-origin-installer-e2e-azure-shared-vpc-4.7 - 1 runs, 100% failed, 100% of failures match

Version-Release number of selected component (if applicable):

4.7

How reproducible:

Rare.

Steps to Reproduce:

1. Install a bunch of clusters.

Actual results:

See some of them whose marketplace operator is not setting Degraded.

Expected results:

Always have the Degraded condition set (ideally with status False)

Additional info:

Seems related to bug 1857680, but the logic discussed in that bug no longer exists in 4.7.

Comment 1 W. Trevor King 2020-11-09 20:45:44 UTC
Example job [1]:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1325575034390974464/artifacts/e2e-aws/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "marketplace").status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")'
2020-11-08T23:23:55Z Progressing=False OperatorAvailable: Successfully progressed to release version: 4.7.0-0.nightly-2020-11-08-225909
2020-11-08T23:23:55Z Available=True OperatorAvailable: Available release version: 4.7.0-0.nightly-2020-11-08-225909

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1325575034390974464

Comment 4 Jian Zhang 2020-11-11 06:43:27 UTC
LGTM, verify it, example job[1]:

[root@preserve-olm-env data]# curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1326370102743928832/artifacts/e2e-aws/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "marketplace").status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")'
2020-11-11T04:03:38Z Progressing=False OperatorAvailable: Successfully progressed to release version: 4.7.0-0.nightly-2020-11-11-033756
2020-11-11T04:03:38Z Available=True OperatorAvailable: Available release version: 4.7.0-0.nightly-2020-11-11-033756
2020-11-11T04:03:38Z Upgradeable=True OperatorAvailable: Marketplace is upgradeable
2020-11-11T04:03:17Z Degraded=False OperatorAvailable: Successfully progressed to release version: 4.7.0-0.nightly-2020-11-11-033756

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1326370102743928832

Comment 7 errata-xmlrpc 2021-02-24 15:31:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.