Bug 1688656 - [cloud] clusteroperator machine-api didn't update lastTransitionTime field after upgrade
Summary: [cloud] clusteroperator machine-api didn't update lastTransitionTime field af...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.1.0
Assignee: Michael Gugino
QA Contact: sunzhaohua
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-14 07:38 UTC by sunzhaohua
Modified: 2019-06-04 10:45 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:45:49 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:45:56 UTC

Description sunzhaohua 2019-03-14 07:38:25 UTC
Description of problem:
After upgrade OCP 4.0, the clusteroperator machine-api didn't update lastTransitionTime field

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-04-234414   True        False         21m     Cluster version is 4.0.0-0.nightly-2019-03-04-234414

How reproducible:
Always

Steps to Reproduce:
1. Install a cluster 4.0 with 4.0.0-0.nightly-2019-03-04-234414 version.
2. Check the clusteroperator machine-api status, as below:
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-14T02:59:46Z
  generation: 1
  name: machine-api
  resourceVersion: "21449"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-api
  uid: 39cc3c3d-4605-11e9-85a3-023bb53f2bf0
spec: {}
status:
  conditions:
  - lastTransitionTime: null
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-14T02:59:47Z
    message: 'Cluster Machine API Operator is available at operator: 4.0.0-0.nightly-2019-03-04-234414'
    status: "True"
    type: Available
  - lastTransitionTime: 2019-03-14T02:59:47Z
    status: "False"
    type: Failing
  extension: null
  relatedObjects: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-03-04-234414
    
3. Upgrade the cluster
4. Check the status of clusteroperator machine-api status.

Actual results:
After upgrade the version info was updated, the status->conditions->lastTransitionTime didn't get updated.

apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-14T02:59:46Z
  generation: 1
  name: machine-api
  resourceVersion: "33053"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-api
  uid: 39cc3c3d-4605-11e9-85a3-023bb53f2bf0
spec: {}
status:
  conditions:
  - lastTransitionTime: null
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-14T02:59:47Z
    message: 'Cluster Machine API Operator is available at operator: 4.0.0-0.nightly-2019-03-05-045224'
    status: "True"
    type: Available
  - lastTransitionTime: 2019-03-14T02:59:47Z
    status: "False"
    type: Failing
  extension: null
  relatedObjects: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-03-05-045224

Expected results:
After upgrade, the status->conditions->lastTransitionTime should be updated, too.

Additional info:

Comment 1 sunzhaohua 2019-03-15 09:58:40 UTC
>   relatedObjects: null

The related objects should NOT be null.

Comment 2 Michael Gugino 2019-03-29 14:23:20 UTC
This is the result of:

openshift/machine-api-operator/vendor/github.com/openshift/cluster-version-operator/lib/resourcemerge/os.go

This library only compares previous condition with current condition.  In the case of machine-api, we are always status=available during upgrade as we're not checking an operand yet.

Comment 3 Michael Gugino 2019-03-29 15:28:24 UTC
Looking into this a bit more, it appears we don't update the operator version until after we wait for deployment rollout of new version, even though we don't 'check' the operand version directly.

Unsure what the root cause is at this time.  Do you happen to have machine-api-operator verbose logs?

Comment 4 Michael Gugino 2019-04-02 01:31:32 UTC
Proposed fix to update transition time for progressing:true -> progressing:false https://github.com/openshift/cluster-version-operator/pull/153

Will need specific change for available transition time after update, if desired.

Comment 5 sunzhaohua 2019-04-12 09:26:06 UTC
Verified.

Upgrade from 4.0.0-0.nightly-2019-04-10-141956 to 4.0.0-0.nightly-2019-04-10-182914

Before upgrade:
$ oc get clusteroperator machine-api -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-04-11T02:31:45Z
  generation: 1
  name: machine-api
  resourceVersion: "25972"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-api
  uid: f2d395da-5c01-11e9-a622-02a87c79c5de
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-11T02:32:09Z
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-11T02:31:45Z
    message: 'Cluster Machine API Operator is available at operator: 4.0.0-0.nightly-2019-04-10-141956'
    status: "True"
    type: Available
  - lastTransitionTime: 2019-04-11T02:31:45Z
    status: "False"
    type: Failing
  extension: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-04-10-141956

After upgrade:
$ oc get clusteroperator machine-api -o yaml 
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-04-11T02:31:45Z
  generation: 1
  name: machine-api
  resourceVersion: "122537"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-api
  uid: f2d395da-5c01-11e9-a622-02a87c79c5de
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-11T03:43:13Z
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-11T02:31:45Z
    message: 'Cluster Machine API Operator is available at operator: 4.0.0-0.nightly-2019-04-10-182914'
    status: "True"
    type: Available
  - lastTransitionTime: 2019-04-11T02:31:45Z
    status: "False"
    type: Failing
  extension: null
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-04-10-182914

Comment 7 errata-xmlrpc 2019-06-04 10:45:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.