Bug 1857680
| Summary: | The marketplace co should recover after other cluserOperators work well when the success ratio meets 0.3 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jian Zhang <jiazha> |
| Component: | OLM | Assignee: | Alexander Greene <agreene> |
| OLM sub component: | OperatorHub | QA Contact: | Jian Zhang <jiazha> |
| Status: | CLOSED CANTFIX | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | dageoffr, krizza, nhale, wking |
| Version: | 4.5 | Keywords: | Reopened |
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-22 16:30:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1881540 | ||
Hi Kevin, We still get this issue when installing the 4.5.0-0.nightly-2020-09-16-205243. [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 6h44m Unable to apply 4.5.0-0.nightly-2020-09-16-205243: the cluster operator marketplace has not yet successfully rolled out [root@preserve-olm-env data]# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.0-0.nightly-2020-09-16-205243 True False False 6h6m cloud-credential 4.5.0-0.nightly-2020-09-16-205243 True False False 6h26m cluster-autoscaler 4.5.0-0.nightly-2020-09-16-205243 True False False 6h16m config-operator 4.5.0-0.nightly-2020-09-16-205243 True False False 6h16m console 4.5.0-0.nightly-2020-09-16-205243 True False False 6h3m csi-snapshot-controller 4.5.0-0.nightly-2020-09-16-205243 True False False 71m dns 4.5.0-0.nightly-2020-09-16-205243 True False False 6h23m etcd 4.5.0-0.nightly-2020-09-16-205243 True False False 6h22m image-registry 4.5.0-0.nightly-2020-09-16-205243 True False False 6h12m ingress 4.5.0-0.nightly-2020-09-16-205243 True False False 6h12m insights 4.5.0-0.nightly-2020-09-16-205243 True False False 6h17m kube-apiserver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h21m kube-controller-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h22m kube-scheduler 4.5.0-0.nightly-2020-09-16-205243 True False False 6h22m kube-storage-version-migrator 4.5.0-0.nightly-2020-09-16-205243 True False False 2m20s machine-api 4.5.0-0.nightly-2020-09-16-205243 True False False 6h13m machine-approver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h21m machine-config 4.5.0-0.nightly-2020-09-16-205243 True False False 113s marketplace False monitoring 4.5.0-0.nightly-2020-09-16-205243 False True True 9s network 4.5.0-0.nightly-2020-09-16-205243 True False True 6h24m node-tuning 4.5.0-0.nightly-2020-09-16-205243 True False False 6h24m openshift-apiserver 4.5.0-0.nightly-2020-09-16-205243 False False False 20s openshift-controller-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h17m openshift-samples 4.5.0-0.nightly-2020-09-16-205243 True False False 6h16m operator-lifecycle-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h23m operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-09-16-205243 True False False 6h23m operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h19m service-ca 4.5.0-0.nightly-2020-09-16-205243 True False False 6h24m storage 4.5.0-0.nightly-2020-09-16-205243 True False False 6h17m And, after the openshift-apiserver ready, the marketplace still in Failure status. [root@preserve-olm-env data]# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.0-0.nightly-2020-09-16-205243 True False False 6h21m cloud-credential 4.5.0-0.nightly-2020-09-16-205243 True False False 6h41m cluster-autoscaler 4.5.0-0.nightly-2020-09-16-205243 True False False 6h31m config-operator 4.5.0-0.nightly-2020-09-16-205243 True False False 6h31m console 4.5.0-0.nightly-2020-09-16-205243 True False False 6h18m csi-snapshot-controller 4.5.0-0.nightly-2020-09-16-205243 True False False 86m dns 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m etcd 4.5.0-0.nightly-2020-09-16-205243 True False False 6h37m image-registry 4.5.0-0.nightly-2020-09-16-205243 True False False 6h27m ingress 4.5.0-0.nightly-2020-09-16-205243 True False False 6h27m insights 4.5.0-0.nightly-2020-09-16-205243 True False False 6h32m kube-apiserver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h36m kube-controller-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h37m kube-scheduler 4.5.0-0.nightly-2020-09-16-205243 True False False 6h36m kube-storage-version-migrator 4.5.0-0.nightly-2020-09-16-205243 True False False 17m machine-api 4.5.0-0.nightly-2020-09-16-205243 True False False 6h28m machine-approver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h36m machine-config 4.5.0-0.nightly-2020-09-16-205243 True False False 14m marketplace False monitoring 4.5.0-0.nightly-2020-09-16-205243 True False False 12m network 4.5.0-0.nightly-2020-09-16-205243 True False False 6h39m node-tuning 4.5.0-0.nightly-2020-09-16-205243 True False False 6h39m openshift-apiserver 4.5.0-0.nightly-2020-09-16-205243 True False False 13m openshift-controller-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h32m openshift-samples 4.5.0-0.nightly-2020-09-16-205243 True False False 6h31m operator-lifecycle-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h34m service-ca 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m storage 4.5.0-0.nightly-2020-09-16-205243 True False False 6h32m Workaround: delete the pods in the marketplace project [root@preserve-olm-env data]# oc project Using project "openshift-marketplace" on server "https://api.whu45gcp6.qe.gcp.devcluster.openshift.com:6443". [root@preserve-olm-env data]# oc delete pods --all pod "certified-operators-5dd878969d-k7j6q" deleted pod "community-operators-5f797ff5d4-8bkqj" deleted pod "marketplace-operator-68fc669f45-phgmh" deleted pod "redhat-marketplace-744f49cd69-sl2pl" deleted pod "redhat-operators-668696587c-b4lzk" deleted [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-09-16-205243 True False 2m30s Cluster version is 4.5.0-0.nightly-2020-09-16-205243 [root@preserve-olm-env data]# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.0-0.nightly-2020-09-16-205243 True False False 6h28m cloud-credential 4.5.0-0.nightly-2020-09-16-205243 True False False 6h47m cluster-autoscaler 4.5.0-0.nightly-2020-09-16-205243 True False False 6h37m config-operator 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m console 4.5.0-0.nightly-2020-09-16-205243 True False False 6h25m csi-snapshot-controller 4.5.0-0.nightly-2020-09-16-205243 True False False 93m dns 4.5.0-0.nightly-2020-09-16-205243 True False False 6h44m etcd 4.5.0-0.nightly-2020-09-16-205243 True False False 6h43m image-registry 4.5.0-0.nightly-2020-09-16-205243 True False False 6h34m ingress 4.5.0-0.nightly-2020-09-16-205243 True False False 6h33m insights 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m kube-apiserver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h43m kube-controller-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h43m kube-scheduler 4.5.0-0.nightly-2020-09-16-205243 True False False 6h43m kube-storage-version-migrator 4.5.0-0.nightly-2020-09-16-205243 True False False 23m machine-api 4.5.0-0.nightly-2020-09-16-205243 True False False 6h35m machine-approver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h42m machine-config 4.5.0-0.nightly-2020-09-16-205243 True False False 20m marketplace 4.5.0-0.nightly-2020-09-16-205243 True False False 4m54s monitoring 4.5.0-0.nightly-2020-09-16-205243 True False False 19m network 4.5.0-0.nightly-2020-09-16-205243 True False False 6h45m node-tuning 4.5.0-0.nightly-2020-09-16-205243 True False False 6h45m openshift-apiserver 4.5.0-0.nightly-2020-09-16-205243 True False False 20m openshift-controller-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m openshift-samples 4.5.0-0.nightly-2020-09-16-205243 True False False 6h37m operator-lifecycle-manager 4.5.0-0.nightly-2020-09-16-205243 True False False 6h44m operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-09-16-205243 True False False 6h44m operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-09-16-205243 True False False 6h41m service-ca 4.5.0-0.nightly-2020-09-16-205243 True False False 6h45m storage 4.5.0-0.nightly-2020-09-16-205243 True False False 6h38m Reopen it. This bug does not exist in 4.7 because the ratio logic was removed in 4.6 |
Description of problem: Only the "Upgradeable" and "Degraded" status set when the sync ratio > 0.3, lack of the "VERSION AVAILABLE PROGRESSING" conditions set. [root@preserve-olm-env data]# oc get co ... NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE ... machine-config 4.5.2 True False False 13m marketplace False Version-Release number of selected component (if applicable): [root@preserve-olm-env data]# oc exec marketplace-operator-798d596899-6hjs4 -- marketplace-operator --version Marketplace source git commit: eeeee277407370e56b445a9c6630507161bb16d3 time="2020-07-16T10:31:53Z" level=info msg="Go Version: go1.13.4" time="2020-07-16T10:31:53Z" level=info msg="Go OS/Arch: linux/amd64" time="2020-07-16T10:31:53Z" level=info msg="operator-sdk Version: v0.8.0" How reproducible: always Steps to Reproduce: 1. Install OCP 4.5.2. 2. Create some invlide CRs. 3. check the marketplace cluster operator. Actual results: Lack of the "VERSION AVAILABLE PROGRESSING SINCE" conditions status. And, get the errors: [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 47m Unable to apply 4.5.2: the cluster operator marketplace has not yet successfully rolled out [root@preserve-olm-env data]# oc get co marketplace -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: creationTimestamp: "2020-07-16T01:48:36Z" generation: 1 managedFields: - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:spec: {} f:status: .: {} f:extension: {} f:versions: {} manager: cluster-version-operator operation: Update time: "2020-07-16T01:48:37Z" - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: {} f:relatedObjects: {} manager: marketplace-operator operation: Update time: "2020-07-16T02:21:34Z" name: marketplace resourceVersion: "24582" selfLink: /apis/config.openshift.io/v1/clusteroperators/marketplace uid: 45aed17c-8c42-4471-83a2-e44efa6bd0f7 spec: {} status: conditions: - lastTransitionTime: "2020-07-16T02:02:12Z" message: Marketplace is upgradeable reason: OperatorAvailable status: "True" type: Upgradeable - lastTransitionTime: "2020-07-16T02:01:52Z" message: Current CR sync ratio (0.9189189) meets the expected success ratio (0.3) reason: OperandTransitionsSucceeding status: "False" type: Degraded extension: null relatedObjects: - group: "" name: openshift-marketplace resource: namespaces - group: operators.coreos.com name: "" namespace: openshift-marketplace resource: OperatorSource - group: operators.coreos.com name: "" namespace: openshift-marketplace resource: CatalogSource Expected results: All conditions should be set. Additional info: I think the root casue is here: https://github.com/operator-framework/operator-marketplace/blob/release-4.5/pkg/status/status.go#L339-L353