Bug 1759712

Summary:	ClusterVersion upgrade history of previously completed upgrades completionTimestamp is being modified when it shouldnt
Product:	OpenShift Container Platform	Reporter:	W. Trevor King <wking>
Component:	Cluster Version Operator	Assignee:	W. Trevor King <wking>
Status:	CLOSED ERRATA	QA Contact:	liujia <jiajliu>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.1.z	CC:	aos-bugs, ccoleman, chancez, jiajliu, jokerman, rszumski, wking
Target Milestone:	---
Target Release:	4.1.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1756461	Environment:
Last Closed:	2019-10-30 19:01:18 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1759710
Bug Blocks:

Description W. Trevor King 2019-10-08 21:38:23 UTC

+++ This bug was initially created as a clone of Bug #1756461 +++

Description of problem: The ClusterVersion object in my cluster is having previously completed upgrade history modified post-upgrade such that it no longer is providing an accurate summary of upgrade history. I am seeing the timestamps on upgrades getting aligned end to end (upgrade completed time is always equal to the next upgrade's startedTime) and I know from previously looking at upgrades that this isn't accurate, and I know the values were not always like this, hence the conclusion that the history is being modified post-upgrade.


Version-Release number of selected component (if applicable): 4.1.18


How reproducible: Seems common. Rob Szumski also hit this.


Steps to Reproduce:
1. Run upgrades
2. Check Openshift UI
3. Wait
4. Do another upgrade
5. Verify even though you waited, the first upgrade's completedTime will equal the 2nd upgrades startedTime.

Actual results: Incorrect upgrade history times


Expected results: Correct upgrade history times


Additional info:

ClusterID is af8bc55b-9ae3-4735-bf65-b6ef43aeced9

You can see a snippet of my previous upgrade history in the following ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1753348 

In particular, this was my latest upgrades information from the ClusterVersion history list 4.1.16. Pay attention to completionTime.

    history:
    - completionTime: "2019-09-14T01:25:04Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:61ed953962d43cae388cb3c544b4cac358d4675076c2fc0befb236209d5116f7
      startedTime: "2019-09-14T00:03:21Z"
      state: Completed
      verified: true
      version: 4.1.16


Now, look at the same information taken today from the same cluster at 4.1.18:

    history:
    - completionTime: "2019-09-25T17:22:57Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:420633acf3fc7572372fe2df758152f6ab1f53a21c79a6c4b741fa0394c7df3a
      startedTime: "2019-09-25T16:13:14Z"
      state: Completed
      verified: true
      version: 4.1.18
    - completionTime: "2019-09-25T16:13:14Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:747e0d41ee2f1af8b234e8c96c3291225a120fab3af53ae691afb4f51ce02b85
      startedTime: "2019-09-23T23:01:54Z"
      state: Completed
      verified: true
      version: 4.1.17
    - completionTime: "2019-09-23T23:01:54Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:61ed953962d43cae388cb3c544b4cac358d4675076c2fc0befb236209d5116f7
      startedTime: "2019-09-14T00:03:21Z"
      state: Completed
      verified: true
      version: 4.1.16

Notice that the version 4.1.16 completionTime was:
    - completionTime: "2019-09-14T01:25:04Z"

and the current value of that same item in the CV history list's completionTime for 4.1.16 is:
    - completionTime: "2019-09-23T23:01:54Z"


As shown, the completionTime of a previously, 2 versions ago upgrade changed.

Comment 1 W. Trevor King 2019-10-08 22:37:52 UTC

This should not block the 4.2.z bug 1759710.  Removing that.

Comment 2 W. Trevor King 2019-10-21 18:11:27 UTC

Changing the blocker back to the 4.3.0 bug 1756461 to test Bugzilla bot handling.

Comment 3 W. Trevor King 2019-10-21 18:14:22 UTC

Changing back to the 4.2.z bug 1759710, since we don't want to fix in 4.1.z before we fix in 4.2.z.

Comment 5 liujia 2019-10-24 07:14:34 UTC

Version:
4.1.0-0.nightly-2019-10-23-020857

before upgrade
# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2019-10-24T04:01:36Z|2019-10-24T04:14:52Z|Completed|4.1.0-0.nightly-2019-10-23-020857

after upgrade
# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2019-10-24T07:07:37Z||Partial|4.1.0-0.nightly-2019-10-23-030305
2019-10-24T07:01:07Z|2019-10-24T07:01:22Z|Completed|4.1.0-0.nightly-2019-10-23-020857
2019-10-24T07:00:52Z|2019-10-24T07:01:07Z|Partial|
2019-10-24T04:01:36Z|2019-10-24T04:14:52Z|Completed|4.1.0-0.nightly-2019-10-23-020857

Comment 7 errata-xmlrpc 2019-10-30 19:01:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3152