https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.2/20/

During an upgrade rollback test (start upgrade, then abort and rollback) the network operator appears to have hung.  Needs triage (we need to explicitly know whether upgrades can rollback, and this sort of failure has to be eliminated / understood).

May 26 21:48:35.102: INFO: Upgrade will be aborted and the cluster will roll back to the current version after 87% of operators have upgraded (picked randomly)
May 26 21:48:40.177: INFO: Cluster version operator acknowledged upgrade request
May 26 21:55:50.305: INFO: cluster upgrade is failing: Multiple errors are preventing progress:
* Could not update deployment "openshift-cloud-credential-operator/cloud-credential-operator" (93 of 371)
* Could not update deployment "openshift-operator-lifecycle-manager/packageserver" (269 of 371)
May 26 21:56:00.249: INFO: cluster upgrade is failing: Multiple errors are preventing progress:
* Could not update deployment "openshift-cloud-credential-operator/cloud-credential-operator" (93 of 371)
* Could not update deployment "openshift-operator-lifecycle-manager/packageserver" (269 of 371)
May 26 21:56:10.254: INFO: cluster upgrade is failing: Multiple errors are preventing progress:
* Could not update deployment "openshift-cloud-credential-operator/cloud-credential-operator" (93 of 371)
* Could not update deployment "openshift-operator-lifecycle-manager/packageserver" (269 of 371)
May 26 21:58:10.285: INFO: -------------------------------------------------------
May 26 21:58:10.285: INFO: Upgraded 23/26 operators, beginning controlled rollback
May 26 21:58:10.285: INFO: Instructing the cluster to return to 4.2.0-0.ci-2019-05-24-200146 / registry.svc.ci.openshift.org/ocp/release@sha256:67594d83cd9b48139fb48ed994c37a689e2cd317dbc6ad1e3c4ecaeff256ae6f
May 26 22:24:00.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:10.251: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:20.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:30.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:40.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:24:50.249: INFO: cluster upgrade is failing: Could not update deployment "openshift-machine-api/cluster-autoscaler-operator" (123 of 371)
May 26 22:33:50.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:00.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:10.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:20.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:30.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:40.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:34:50.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:00.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:10.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:20.250: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:35:30.249: INFO: cluster upgrade is failing: Cluster operator network is still updating
May 26 22:42:20.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:42:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:42:40.254: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:42:50.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:00.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:10.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:20.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:40.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:43:50.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:00.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:10.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:20.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:40.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:44:50.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:00.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:10.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:20.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:30.249: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:45:40.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: cluster-autoscaler, network
May 26 22:52:30.250: INFO: cluster upgrade is failing: Some cluster operators are still updating: network, openshift-apiserver