Bug 1759712 - ClusterVersion upgrade history of previously completed upgrades completionTimestamp is being modified when it shouldnt
Summary: ClusterVersion upgrade history of previously completed upgrades completionTim...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.1.z
Assignee: W. Trevor King
QA Contact: liujia
URL:
Whiteboard:
Depends On: 1759710
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-08 21:38 UTC by W. Trevor King
Modified: 2019-10-30 19:01 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1756461
Environment:
Last Closed: 2019-10-30 19:01:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 251 0 None closed Bug 1759712: pkg/cvo/status: Don't clobber last.CompletionTime when adding new entries 2020-03-18 05:05:02 UTC
Red Hat Product Errata RHBA-2019:3152 0 None None None 2019-10-30 19:01:29 UTC

Description W. Trevor King 2019-10-08 21:38:23 UTC
+++ This bug was initially created as a clone of Bug #1756461 +++

Description of problem: The ClusterVersion object in my cluster is having previously completed upgrade history modified post-upgrade such that it no longer is providing an accurate summary of upgrade history. I am seeing the timestamps on upgrades getting aligned end to end (upgrade completed time is always equal to the next upgrade's startedTime) and I know from previously looking at upgrades that this isn't accurate, and I know the values were not always like this, hence the conclusion that the history is being modified post-upgrade.


Version-Release number of selected component (if applicable): 4.1.18


How reproducible: Seems common. Rob Szumski also hit this.


Steps to Reproduce:
1. Run upgrades
2. Check Openshift UI
3. Wait
4. Do another upgrade
5. Verify even though you waited, the first upgrade's completedTime will equal the 2nd upgrades startedTime.

Actual results: Incorrect upgrade history times


Expected results: Correct upgrade history times


Additional info:

ClusterID is af8bc55b-9ae3-4735-bf65-b6ef43aeced9

You can see a snippet of my previous upgrade history in the following ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1753348 

In particular, this was my latest upgrades information from the ClusterVersion history list 4.1.16. Pay attention to completionTime.

    history:
    - completionTime: "2019-09-14T01:25:04Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:61ed953962d43cae388cb3c544b4cac358d4675076c2fc0befb236209d5116f7
      startedTime: "2019-09-14T00:03:21Z"
      state: Completed
      verified: true
      version: 4.1.16


Now, look at the same information taken today from the same cluster at 4.1.18:

    history:
    - completionTime: "2019-09-25T17:22:57Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:420633acf3fc7572372fe2df758152f6ab1f53a21c79a6c4b741fa0394c7df3a
      startedTime: "2019-09-25T16:13:14Z"
      state: Completed
      verified: true
      version: 4.1.18
    - completionTime: "2019-09-25T16:13:14Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:747e0d41ee2f1af8b234e8c96c3291225a120fab3af53ae691afb4f51ce02b85
      startedTime: "2019-09-23T23:01:54Z"
      state: Completed
      verified: true
      version: 4.1.17
    - completionTime: "2019-09-23T23:01:54Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:61ed953962d43cae388cb3c544b4cac358d4675076c2fc0befb236209d5116f7
      startedTime: "2019-09-14T00:03:21Z"
      state: Completed
      verified: true
      version: 4.1.16

Notice that the version 4.1.16 completionTime was:
    - completionTime: "2019-09-14T01:25:04Z"

and the current value of that same item in the CV history list's completionTime for 4.1.16 is:
    - completionTime: "2019-09-23T23:01:54Z"


As shown, the completionTime of a previously, 2 versions ago upgrade changed.

Comment 1 W. Trevor King 2019-10-08 22:37:52 UTC
This should not block the 4.2.z bug 1759710.  Removing that.

Comment 2 W. Trevor King 2019-10-21 18:11:27 UTC
Changing the blocker back to the 4.3.0 bug 1756461 to test Bugzilla bot handling.

Comment 3 W. Trevor King 2019-10-21 18:14:22 UTC
Changing back to the 4.2.z bug 1759710, since we don't want to fix in 4.1.z before we fix in 4.2.z.

Comment 5 liujia 2019-10-24 07:14:34 UTC
Version:
4.1.0-0.nightly-2019-10-23-020857

before upgrade
# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2019-10-24T04:01:36Z|2019-10-24T04:14:52Z|Completed|4.1.0-0.nightly-2019-10-23-020857

after upgrade
# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2019-10-24T07:07:37Z||Partial|4.1.0-0.nightly-2019-10-23-030305
2019-10-24T07:01:07Z|2019-10-24T07:01:22Z|Completed|4.1.0-0.nightly-2019-10-23-020857
2019-10-24T07:00:52Z|2019-10-24T07:01:07Z|Partial|
2019-10-24T04:01:36Z|2019-10-24T04:14:52Z|Completed|4.1.0-0.nightly-2019-10-23-020857

Comment 7 errata-xmlrpc 2019-10-30 19:01:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3152


Note You need to log in before you can comment on or make changes to this bug.