1857680 – The marketplace co should recover after other cluserOperators work well when the success ratio meets 0.3

Bug 1857680 - The marketplace co should recover after other cluserOperators work well when the success ratio meets 0.3

Summary: The marketplace co should recover after other cluserOperators work well when ...

Keywords:
Status:	CLOSED CANTFIX
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	OLM
Sub Component:
Version:	4.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.7.0
Assignee:	Alexander Greene
QA Contact:	Jian Zhang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1881540
TreeView+	depends on / blocked

Reported:	2020-07-16 10:39 UTC by Jian Zhang
Modified:	2020-11-19 18:12 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-09-22 16:30:23 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Jian Zhang 2020-07-16 10:39:42 UTC

Description of problem:
Only the "Upgradeable" and "Degraded" status set when the sync ratio > 0.3, lack of the "VERSION   AVAILABLE   PROGRESSING" conditions set.

[root@preserve-olm-env data]# oc get co
...
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
...
machine-config                             4.5.2     True        False         False      13m
marketplace                                                                    False  

Version-Release number of selected component (if applicable):
[root@preserve-olm-env data]# oc exec marketplace-operator-798d596899-6hjs4 -- marketplace-operator --version
Marketplace source git commit: eeeee277407370e56b445a9c6630507161bb16d3
time="2020-07-16T10:31:53Z" level=info msg="Go Version: go1.13.4"
time="2020-07-16T10:31:53Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-07-16T10:31:53Z" level=info msg="operator-sdk Version: v0.8.0"

How reproducible:
always

Steps to Reproduce:
1. Install OCP 4.5.2.
2. Create some invlide CRs.
3. check the marketplace cluster operator.

Actual results:
Lack of the "VERSION   AVAILABLE   PROGRESSING SINCE" conditions status. And, get the errors:
[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          47m     Unable to apply 4.5.2: the cluster operator marketplace has not yet successfully rolled out

[root@preserve-olm-env data]# oc get co marketplace -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2020-07-16T01:48:36Z"
  generation: 1
  managedFields:
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec: {}
      f:status:
        .: {}
        f:extension: {}
        f:versions: {}
    manager: cluster-version-operator
    operation: Update
    time: "2020-07-16T01:48:37Z"
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions: {}
        f:relatedObjects: {}
    manager: marketplace-operator
    operation: Update
    time: "2020-07-16T02:21:34Z"
  name: marketplace
  resourceVersion: "24582"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/marketplace
  uid: 45aed17c-8c42-4471-83a2-e44efa6bd0f7
spec: {}
status:
  conditions:
  - lastTransitionTime: "2020-07-16T02:02:12Z"
    message: Marketplace is upgradeable
    reason: OperatorAvailable
    status: "True"
    type: Upgradeable
  - lastTransitionTime: "2020-07-16T02:01:52Z"
    message: Current CR sync ratio (0.9189189) meets the expected success ratio (0.3)
    reason: OperandTransitionsSucceeding
    status: "False"
    type: Degraded
  extension: null
  relatedObjects:
  - group: ""
    name: openshift-marketplace
    resource: namespaces
  - group: operators.coreos.com
    name: ""
    namespace: openshift-marketplace
    resource: OperatorSource
  - group: operators.coreos.com
    name: ""
    namespace: openshift-marketplace
    resource: CatalogSource


Expected results:
All conditions should be set. 

Additional info:
I think the root casue is here: https://github.com/operator-framework/operator-marketplace/blob/release-4.5/pkg/status/status.go#L339-L353

Comment 5 Jian Zhang 2020-09-17 10:30:57 UTC

Hi Kevin,

We still get this issue when installing the 4.5.0-0.nightly-2020-09-16-205243.

[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          6h44m   Unable to apply 4.5.0-0.nightly-2020-09-16-205243: the cluster operator marketplace has not yet successfully rolled out

[root@preserve-olm-env data]# oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h6m
cloud-credential                           4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h26m
cluster-autoscaler                         4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h16m
config-operator                            4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h16m
console                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h3m
csi-snapshot-controller                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      71m
dns                                        4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h23m
etcd                                       4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h22m
image-registry                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h12m
ingress                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h12m
insights                                   4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h17m
kube-apiserver                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h21m
kube-controller-manager                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h22m
kube-scheduler                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h22m
kube-storage-version-migrator              4.5.0-0.nightly-2020-09-16-205243   True        False         False      2m20s
machine-api                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h13m
machine-approver                           4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h21m
machine-config                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      113s
marketplace                                                                                              False      
monitoring                                 4.5.0-0.nightly-2020-09-16-205243   False       True          True       9s
network                                    4.5.0-0.nightly-2020-09-16-205243   True        False         True       6h24m
node-tuning                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h24m
openshift-apiserver                        4.5.0-0.nightly-2020-09-16-205243   False       False         False      20s
openshift-controller-manager               4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h17m
openshift-samples                          4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h16m
operator-lifecycle-manager                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h23m
operator-lifecycle-manager-catalog         4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h23m
operator-lifecycle-manager-packageserver   4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h19m
service-ca                                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h24m
storage                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h17m


And, after the openshift-apiserver ready, the marketplace still in Failure status.
[root@preserve-olm-env data]# oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h21m
cloud-credential                           4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h41m
cluster-autoscaler                         4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h31m
config-operator                            4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h31m
console                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h18m
csi-snapshot-controller                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      86m
dns                                        4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
etcd                                       4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h37m
image-registry                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h27m
ingress                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h27m
insights                                   4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h32m
kube-apiserver                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h36m
kube-controller-manager                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h37m
kube-scheduler                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h36m
kube-storage-version-migrator              4.5.0-0.nightly-2020-09-16-205243   True        False         False      17m
machine-api                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h28m
machine-approver                           4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h36m
machine-config                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      14m
marketplace                                                                                              False      
monitoring                                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      12m
network                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h39m
node-tuning                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h39m
openshift-apiserver                        4.5.0-0.nightly-2020-09-16-205243   True        False         False      13m
openshift-controller-manager               4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h32m
openshift-samples                          4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h31m
operator-lifecycle-manager                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
operator-lifecycle-manager-catalog         4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
operator-lifecycle-manager-packageserver   4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h34m
service-ca                                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
storage                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h32m

Workaround: delete the pods in the marketplace project 
[root@preserve-olm-env data]# oc project
Using project "openshift-marketplace" on server "https://api.whu45gcp6.qe.gcp.devcluster.openshift.com:6443".

[root@preserve-olm-env data]# oc delete pods --all
pod "certified-operators-5dd878969d-k7j6q" deleted
pod "community-operators-5f797ff5d4-8bkqj" deleted
pod "marketplace-operator-68fc669f45-phgmh" deleted
pod "redhat-marketplace-744f49cd69-sl2pl" deleted
pod "redhat-operators-668696587c-b4lzk" deleted

[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-09-16-205243   True        False         2m30s   Cluster version is 4.5.0-0.nightly-2020-09-16-205243
[root@preserve-olm-env data]# oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h28m
cloud-credential                           4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h47m
cluster-autoscaler                         4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h37m
config-operator                            4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
console                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h25m
csi-snapshot-controller                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      93m
dns                                        4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h44m
etcd                                       4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h43m
image-registry                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h34m
ingress                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h33m
insights                                   4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
kube-apiserver                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h43m
kube-controller-manager                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h43m
kube-scheduler                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h43m
kube-storage-version-migrator              4.5.0-0.nightly-2020-09-16-205243   True        False         False      23m
machine-api                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h35m
machine-approver                           4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h42m
machine-config                             4.5.0-0.nightly-2020-09-16-205243   True        False         False      20m
marketplace                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      4m54s
monitoring                                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      19m
network                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h45m
node-tuning                                4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h45m
openshift-apiserver                        4.5.0-0.nightly-2020-09-16-205243   True        False         False      20m
openshift-controller-manager               4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m
openshift-samples                          4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h37m
operator-lifecycle-manager                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h44m
operator-lifecycle-manager-catalog         4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h44m
operator-lifecycle-manager-packageserver   4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h41m
service-ca                                 4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h45m
storage                                    4.5.0-0.nightly-2020-09-16-205243   True        False         False      6h38m

Reopen it.

Comment 7 Alexander Greene 2020-09-22 16:30:23 UTC

This bug does not exist in 4.7 because the ratio logic was removed in 4.6

Note You need to log in before you can comment on or make changes to this bug.