Bug 1874093

Summary:	Unable to apply upgrade in disconnected env - OCP4.5.7 to OCP4.6 nightly without force flag
Product:	OpenShift Container Platform	Reporter:	Shelly Miron <smiron>
Component:	Installer	Assignee:	Beth White <beth.white>
Installer sub component:	OpenShift on Bare Metal IPI	QA Contact:	Amit Ugol <augol>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	unspecified	Keywords:	TestBlocker, Upgrades
Version:	4.6
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-09-01 16:38:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Shelly Miron 2020-08-31 14:32:19 UTC

Description of problem:
----------------------------

Upgrade ran on a disconnected cluster from version 4.5.7 to version 4.6 nightly without using force flag.
I let the upgrade run for some time, but it seems like it stucked with this message:

[kni@provisionhost-0-0 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.7     True        True          3h49m   Unable to apply 4.6.0-0.nightly-2020-08-26-032807: the update could not be applied

[kni@provisionhost-0-0 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-08-26-032807   False       False         True       19s
cloud-credential                           4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h44m
cluster-autoscaler                         4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h20m
config-operator                            4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h21m
console                                    4.6.0-0.nightly-2020-08-26-032807   True        False         True       3h26m
csi-snapshot-controller                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h4m
dns                                        4.5.7                               True        False         False      5h27m
etcd                                       4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h27m
image-registry                             4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h21m
ingress                                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h26m
insights                                   4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h25m
kube-apiserver                             4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h26m
kube-controller-manager                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h26m
kube-scheduler                             4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h23m
kube-storage-version-migrator              4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h4m
machine-api                                4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h17m
machine-approver                           4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h26m
machine-config                             4.5.7                               True        False         False      5h27m
marketplace                                4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h26m
monitoring                                 4.6.0-0.nightly-2020-08-26-032807   False       False         True       34m
network                                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h28m
node-tuning                                4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h26m
openshift-apiserver                        4.6.0-0.nightly-2020-08-26-032807   False       False         False      13m
openshift-controller-manager               4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h23m
openshift-samples                          4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h17m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h28m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h28m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-08-26-032807   False       True          False      51s
service-ca                                 4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h28m
storage                                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h27m


Around 20 minutes after been stuck in this state ( for 100 minutes ), the cluster tried again to upgrade but arrive to the same state.

[kni@provisionhost-0-0 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.7     True        True          4h3m    Working towards 4.6.0-0.nightly-2020-08-26-032807: 1% complete


Operators condition:
------------------------
[kni@provisionhost-0-0 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-08-26-032807   False       False         True       1s
cloud-credential                           4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h59m
cluster-autoscaler                         4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h34m
config-operator                            4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h35m
console                                    4.6.0-0.nightly-2020-08-26-032807   True        False         True       3h40m
csi-snapshot-controller                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h18m
dns                                        4.5.7                               True        False         False      5h41m
etcd                                       4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h41m
image-registry                             4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h35m
ingress                                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h40m
insights                                   4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h39m
kube-apiserver                             4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h41m
kube-controller-manager                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h41m
kube-scheduler                             4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h38m
kube-storage-version-migrator              4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h18m
machine-api                                4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h31m
machine-approver                           4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h40m
machine-config                             4.5.7                               True        False         False      5h41m
marketplace                                4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h40m
monitoring                                 4.6.0-0.nightly-2020-08-26-032807   False       False         True       48m
network                                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h42m
node-tuning                                4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h40m
openshift-apiserver                        4.6.0-0.nightly-2020-08-26-032807   False       False         False      3m59s
openshift-controller-manager               4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h37m
openshift-samples                          4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h32m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h42m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h42m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-08-26-032807   False       True          False      11m
service-ca                                 4.6.0-0.nightly-2020-08-26-032807   True        False         False      5h42m
storage                                    4.6.0-0.nightly-2020-08-26-032807   True        False         False      3h41m



How reproducible:
----------------------
2 out of 2 failed with the same steps, so i would say always.


Steps to Reproduce:
----------------------

1. Deploy disconnected cluster with IPV6 provisioning network and IPV4 baremetal network,
   with version 4.5.7 : https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasetag/4.5.7

2.Upgrade the cluster to version 4.6 nightly ( i used this one: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasetag/4.6.0-0.nightly-2020-08-26-010422 or this one : https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasetag/4.6.0-0.nightly-2020-08-26-032807 ) without using force flag
  
  $ oc adm upgrade --to 4.6.0-0.nightly-2020-08-26-032807



Actual results:
----------------------
Cluster failed to upgrade to version 4.6 nightly after a long period of time.


Expected results:
----------------------
Upgrade from 4.5.7 to 4.6 nightly passed successfully.


Additional info:
----------------------

must- gather: could not add must-gather due to internal error in the cluster;

Comment 1 Beth White 2020-09-01 16:38:23 UTC


*** This bug has been marked as a duplicate of bug 1872742 ***