Bug 1715106

Summary:	Rollback of an update on a 4.2 -> 4.2 nightly failed on network operator (never made progress)
Product:	OpenShift Container Platform	Reporter:	Clayton Coleman <ccoleman>
Component:	Networking	Assignee:	Casey Callendrello <cdc>
Status:	CLOSED DUPLICATE	QA Contact:	zhaozhanqi <zzhao>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.2.0	CC:	aos-bugs, bbennett
Target Milestone:	---
Target Release:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-18 15:33:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Clayton Coleman 2019-05-29 15:13:06 UTC

https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.2/20/

During an upgrade rollback test (start upgrade, then abort and rollback) the network operator appears to have hung.  Needs triage (we need to explicitly know whether upgrades can rollback, and this sort of failure has to be eliminated / understood).

May 26 21:48:35.102: INFO: Upgrade will be aborted and the cluster will roll back to the current version after 87% of operators have upgraded (picked randomly)
May 26 21:48:40.177: INFO: Cluster version operator acknowledged upgrade request
May 26 21:55:50.305: INFO: cluster upgrade is failing: Multiple errors are preventing progress:
* Could not update deployment "openshift-cloud-credential-operator/cloud-credential-operator" (93 of 371)
* Could not update deployment "openshift-operator-lifecycle-manager/packageserver" (269 of 371)
May 26 21:56:00.249: INFO: cluster upgrade is failing: Multiple errors are preventing progress:
* Could not update deployment "openshift-cloud-credential-operator/cloud-credential-operator" (93 of 371)
* Could not update deployment "openshift-operator-lifecycle-manager/packageserver" (269 of 371)
May 26 21:56:10.254: INFO: cluster upgrade is failing: Multiple errors are preventing progress:
* Could not update deployment "openshift-cloud-credential-operator/cloud-credential-operator" (93 of 371)
* Could not update deployment "openshift-operator-lifecycle-manager/packageserver" (269 of 371)
May 26 21:58:10.285: INFO: -------------------------------------------------------
May 26 21:58:10.285: INFO: Upgraded 23/26 operators, beginning controlled rollback
May 26 21:58:10.285: INFO: Instructing the cluster to return to 4.2.0-0.ci-2019-05-24-200146 / registry.svc.ci.openshift.org/ocp/release@sha256:67594d83cd9b48139fb48ed994c37a689e2cd317dbc6ad1e3c4ecaeff256ae6f
May 26 22:24:00.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:10.251: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:20.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:30.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:40.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:50.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:33:50.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:00.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:10.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:20.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:30.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:40.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:50.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:00.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:10.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:20.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:30.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:42:20.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:42:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:42:40.254: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:42:50.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:00.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:10.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:20.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:40.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:50.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:00.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:10.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:20.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:40.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:50.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:00.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:10.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:20.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:40.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:52:30.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: network, openshift-apiserver

Comment 1 Casey Callendrello 2019-06-03 13:37:15 UTC

I believe this to be a duplicate of 1714699.

Comment 2 Casey Callendrello 2019-06-18 15:33:28 UTC


*** This bug has been marked as a duplicate of bug 1714699 ***