Bug 1688611 - OLM operators are incorrectly reporting status in ClusterOperator
Summary: OLM operators are incorrectly reporting status in ClusterOperator
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Evan Cordell
QA Contact: Bruno Andrade
URL:
Whiteboard:
: 1695198 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-14 05:29 UTC by Jian Zhang
Modified: 2019-06-04 10:45 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:45:49 UTC
Target Upstream Version:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:45:56 UTC

Comment 3 Jian Zhang 2019-03-19 09:20:28 UTC 
OLM version commit: io.openshift.build.commit.id=69457423c2da01da0110b17fac1ac48b994b99e8
Cluster version is 4.0.0-0.nightly-2019-03-18-200009

Before upgrade:
[jzhang@dhcp-140-18 ocp119]$ oc get clusteroperator operator-lifecycle-manager -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-19T07:21:00Z
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "1977"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: 8c2d2ee8-4a17-11e9-b038-02e816726d9a
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-03-19T07:21:00Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-19T07:21:00Z
    message: Deployed 0.8.1
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-19T07:21:00Z
    status: "True"
    type: Available
  extension: null
  relatedObjects: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-03-18-200009
  - name: operator-lifecycle-manager
    version: 0.8.1

After upgrade:
[jzhang@dhcp-140-18 ocp119]$ oc get clusteroperator operator-lifecycle-manager -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-19T07:21:00Z
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "70850"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: 8c2d2ee8-4a17-11e9-b038-02e816726d9a
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-03-19T07:21:00Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-19T07:21:00Z
    message: Deployed 0.8.1
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-19T07:21:00Z
    status: "True"
    type: Available
  extension: null
  relatedObjects: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-03-18-223058
  - name: operator-lifecycle-manager
    version: 0.8.1

We can see the version has been updated, but the timestamps of the "conditions" still not be updated.
And, the "relatedObjects" is null, this is incorrect. Verify failed.

[jzhang@dhcp-140-18 ocp119]$ oc get clusterversion -o yaml|grep history -A 9
    history:
    - completionTime: 2019-03-19T08:52:01Z
      image: registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-18-223058
      startedTime: 2019-03-19T08:29:56Z
      state: Completed
      version: 4.0.0-0.nightly-2019-03-18-223058
    - completionTime: 2019-03-19T08:29:56Z
      image: registry.svc.ci.openshift.org/ocp/release@sha256:e3f2bff3e7a40f7ca0777ada2ad89197a5ab6d7296d3bd12a28dc5aa6b4311dc
      startedTime: 2019-03-19T07:18:49Z
      state: Completed
Comment 4 Evan Cordell 2019-03-21 18:05:41 UTC 
My understanding is that the timestamps should only change if the status has changed. In this case, it sounds like OLM started up and successfully synced before it had to write out conditions, so the timestamps didn't change.

I have a PR out to bump the api clients so that the relatedObjects doesn't show as null: https://github.com/operator-framework/operator-lifecycle-manager/pull/776
Comment 5 Jian Zhang 2019-03-22 02:26:27 UTC 
> My understanding is that the timestamps should only change if the status has changed.

Yes, I think so. For this case, the OLM status/version/image did change, but the timestamps didn't change. I think we should address it.
Comment 7 Evan Cordell 2019-03-29 18:04:44 UTC 
We are reporting status for catalog operator now as well: https://github.com/operator-framework/operator-lifecycle-manager/pull/776
Comment 9 Jian Zhang 2019-04-03 01:45:13 UTC 
*** Bug 1695198 has been marked as a duplicate of this bug. ***
Comment 10 Jian Zhang 2019-04-04 09:05:29 UTC 
Verify failed, the timestamp still no change after upgrading. Details as below:

OLM version:  io.openshift.build.commit.id=9ba3512c5406b62179968e2432b284e9a30c321e
1) Before the upgrade:
[jzhang@dhcp-140-18 444]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-04-02-133735   True        False         3h5m    Cluster version is 4.0.0-0.nightly-2019-04-02-133735

[jzhang@dhcp-140-18 444]$ oc get clusteroperator operator-lifecycle-manager -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-04-04T03:13:36Z
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "1846"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: a2bbfd99-5687-11e9-985b-02f3c2d42294
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-04T03:13:36Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-04-04T03:13:36Z
    message: Deployed 0.8.1
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-04T03:13:36Z
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-04-02-133735
  - name: operator-lifecycle-manager
    version: 0.8.1


[jzhang@dhcp-140-18 444]$ oc get clusteroperator operator-lifecycle-manager-catalog -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-04-04T03:13:40Z
  generation: 1
  name: operator-lifecycle-manager-catalog
  resourceVersion: "2089"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-catalog
  uid: a5440e7f-5687-11e9-985b-02f3c2d42294
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-04T03:13:40Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-04-04T03:13:40Z
    message: Deployed 0.8.1
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-04T03:13:40Z
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-04-02-133735
  - name: operator-lifecycle-manager
    version: 0.8.1

2)After the upgrading:
[jzhang@dhcp-140-18 444]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-04-02-150843   True        False         62m     Cluster version is 4.0.0-0.nightly-2019-04-02-150843
[jzhang@dhcp-140-18 444]$ oc get clusterversion -o yaml|grep history -A 10
    history:
    - completionTime: 2019-04-04T07:58:49Z
      image: registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-04-02-150843
      startedTime: 2019-04-04T07:31:36Z
      state: Completed
      version: 4.0.0-0.nightly-2019-04-02-150843
    - completionTime: 2019-04-04T07:31:36Z
      image: registry.svc.ci.openshift.org/ocp/release@sha256:cf617901c800e0edee8f9875594e44e5633a54b003f8bdbbc60f528a237f321f
      startedTime: 2019-04-04T03:11:25Z
      state: Completed
      version: 4.0.0-0.nightly-2019-04-02-133735

[jzhang@dhcp-140-18 444]$ oc get clusteroperator operator-lifecycle-manager -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-04-04T03:13:36Z
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "95532"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: a2bbfd99-5687-11e9-985b-02f3c2d42294
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-04T03:13:36Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-04-04T03:13:36Z
    message: Deployed 0.8.1
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-04T03:13:36Z
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-04-02-150843
  - name: operator-lifecycle-manager
    version: 0.8.1
[jzhang@dhcp-140-18 444]$ oc get clusteroperator operator-lifecycle-manager-catalog -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-04-04T03:13:40Z
  generation: 1
  name: operator-lifecycle-manager-catalog
  resourceVersion: "95652"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-catalog
  uid: a5440e7f-5687-11e9-985b-02f3c2d42294
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-04T03:13:40Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-04-04T03:13:40Z
    message: Deployed 0.8.1
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-04T03:13:40Z
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-04-02-150843
  - name: operator-lifecycle-manager
    version: 0.8.1
Comment 11 Evan Cordell 2019-04-08 17:56:33 UTC 
Jian, please see previous comments. We do not expect the timestamp to change because the apiserver did not go unavailable long enough to trigger a state transition, and we only mark timestamps for the state transition.
Comment 12 Jian Zhang 2019-04-10 01:32:26 UTC 
Evan,

Thanks for your comments. But, for the timestamp issue, please see bug 1695198 reported by Clayton for more details.
Comment 13 Evan Cordell 2019-04-12 15:27:12 UTC 
https://github.com/operator-framework/operator-lifecycle-manager/pull/811
Comment 14 Jian Zhang 2019-04-25 10:44:12 UTC 
Verify failed. The timeStamp hasn't been changed. As below:

1)Before upgrade:
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-04-24-014305   True        False         25m     Cluster version is 4.1.0-0.nightly-2019-04-24-014305
mac:~ jianzhang$ oc get clusteroperator operator-lifecycle-manager -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-04-25T08:50:15Z"
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "7792"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: 24f7c3a6-6737-11e9-b5dc-0265ca367920
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "False"
    type: Failing
  - lastTransitionTime: null
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-04-24-014305
  - name: operator-lifecycle-manager
    version: 0.9.0

mac:~ jianzhang$ oc get clusteroperator operator-lifecycle-manager-catalog -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-04-25T08:50:15Z"
  generation: 1
  name: operator-lifecycle-manager-catalog
  resourceVersion: "7793"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-catalog
  uid: 24f7f350-6737-11e9-b5dc-0265ca367920
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "False"
    type: Failing
  - lastTransitionTime: null
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-04-24-014305
  - name: operator-lifecycle-manager
    version: 0.9.0


2) After upgrade:
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-04-24-163534   True        False         22m     Cluster version is 4.1.0-0.nightly-2019-04-24-163534

mac:~ jianzhang$ oc get clusteroperator
...
operator-lifecycle-manager           4.1.0-0.nightly-2019-04-24-163534   True        False                    99m
operator-lifecycle-manager-catalog   4.1.0-0.nightly-2019-04-24-163534   True        False                    99m

mac:~ jianzhang$ oc get clusteroperator operator-lifecycle-manager -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-04-25T08:50:15Z"
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "53611"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: 24f7c3a6-6737-11e9-b5dc-0265ca367920
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "False"
    type: Failing
  - lastTransitionTime: null
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-04-24-163534
  - name: operator-lifecycle-manager
    version: 0.9.0
mac:~ jianzhang$ oc get clusteroperator operator-lifecycle-manager-catalog -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-04-25T08:50:15Z"
  generation: 1
  name: operator-lifecycle-manager-catalog
  resourceVersion: "53821"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-catalog
  uid: 24f7f350-6737-11e9-b5dc-0265ca367920
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "False"
    type: Failing
  - lastTransitionTime: null
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-04-25T08:50:15Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-04-24-163534
  - name: operator-lifecycle-manager
    version: 0.9.0

OLM version:
               io.openshift.build.commit.id=b8b77a1db257f5c3dd7a4120118d600bb72916a7
               io.openshift.build.commit.url=https://github.com/operator-framework/operator-lifecycle-manager/commit/b8b77a1db257f5c3dd7a4120118d600bb72916a7
               io.openshift.build.source-location=https://github.com/operator-framework/operator-lifecycle-manager
Comment 15 Maciej Szulik 2019-04-25 15:16:57 UTC 
Additionally, you should bump openshift/api to get https://github.com/openshift/api/pull/287 and react accordingly to that change.
Comment 16 Evan Cordell 2019-04-30 11:07:26 UTC 
Fixed in:

https://github.com/operator-framework/operator-lifecycle-manager/pull/823
https://github.com/operator-framework/operator-lifecycle-manager/pull/830

and verified in the upgrade test:
https://storage.googleapis.com/origin-ci-test/pr-logs/pull/operator-framework_operator-lifecycle-manager/830/pull-ci-operator-framework-operator-lifecycle-manager-master-e2e-aws-upgrade/45/artifacts/e2e-aws-upgrade/clusteroperators.json

    {
            "apiVersion": "config.openshift.io/v1",
            "kind": "ClusterOperator",
            "metadata": {
                "creationTimestamp": "2019-04-29T10:54:03Z",
                "generation": 1,
                "name": "operator-lifecycle-manager",
                "resourceVersion": "28001",
                "selfLink": "/apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager",
                "uid": "19e63429-6a6d-11e9-9ec2-12be92be8bdc"
            },
            "spec": {},
            "status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2019-04-29T10:54:03Z",
                        "status": "False",
                        "type": "Degraded"
                    },
                    {
                        "lastTransitionTime": "2019-04-29T11:27:14Z",
                        "message": "Deployed 0.9.0",
                        "status": "False",
                        "type": "Progressing"
                    },
                    {
                        "lastTransitionTime": "2019-04-29T10:54:03Z",
                        "status": "True",
                        "type": "Available"
                    }
                ],
                "extension": null,
                "versions": [
                    {
                        "name": "operator",
                        "version": "0.0.1-2019-04-29-103147"
                    },
                    {
                        "name": "operator-lifecycle-manager",
                        "version": "0.9.0"
                    }
                ]
            }
        },
Comment 18 Bruno Andrade 2019-05-03 16:07:18 UTC 
Verify succeed, the state transition (Progressing) timestamp was updated as expected.

1) Before update:

oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-02-044125   True        False         3m32s   Cluster version is 4.1.0-0.nightly-2019-05-02-044125


oc get clusteroperator operator-lifecycle-manager -o yaml

apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-05-03T14:13:55Z"
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "12065"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: afae081c-6dad-11e9-89c1-020c072ad952
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2019-05-03T14:19:00Z"
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-05-02-044125
  - name: operator-lifecycle-manager
    version: 0.9.0



oc get clusteroperator operator-lifecycle-manager-catalog -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-05-03T14:13:55Z"
  generation: 1
  name: operator-lifecycle-manager-catalog
  resourceVersion: "12067"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-catalog
  uid: afb3136c-6dad-11e9-89c1-020c072ad952
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2019-05-03T14:19:00Z"
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-05-02-044125
  - name: operator-lifecycle-manager
    version: 0.9.0

2) After upgrade:
oc get clusteroperator operator-lifecycle-manager -o yaml

apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-05-03T14:13:55Z"
  generation: 1
  name: operator-lifecycle-manager
  resourceVersion: "28563"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager
  uid: afae081c-6dad-11e9-89c1-020c072ad952
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2019-05-03T15:07:21Z"
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-05-02-082538
  - name: operator-lifecycle-manager
    version: 0.9.0

oc get clusteroperator operator-lifecycle-manager-catalog -o yaml

apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-05-03T14:13:55Z"
  generation: 1
  name: operator-lifecycle-manager-catalog
  resourceVersion: "28643"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-catalog
  uid: afb3136c-6dad-11e9-89c1-020c072ad952
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2019-05-03T15:07:41Z"
    message: Deployed 0.9.0
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-05-03T14:13:55Z"
    status: "True"
    type: Available
  extension: null
  versions:
  - name: operator
    version: 4.1.0-0.nightly-2019-05-02-082538
  - name: operator-lifecycle-manager
    version: 0.9.0

OLM version:
          "io.openshift.build.commit.id": "bfc2f944d9d7007d445ed64242207bf7ca933f67",
          "io.openshift.build.commit.ref": "",
          "io.openshift.build.source-location": "https://github.com/operator-framework/operator-lifecycle-manager"
Comment 20 errata-xmlrpc 2019-06-04 10:45:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758
Note You need to log in before you can comment on or make changes to this bug.