Description of problem: Performing an upgrade from OCP 4.x to 4.6 will fail if the cluster is currently a restricted network on z/VM. Version-Release number of selected component (if applicable): Upgrades to 4.6.0-rc1 and 4.6.0-rc2 failed. How reproducible: Consistently Steps to Reproduce: 1. Begin with a GA version of OCP cluster installed and operating on z/VM. As a starting point, we had OCP version 4.5.13. 2. Follow the upgrade instructions for restricted network cluster: https://docs.openshift.com/container-platform/4.5/updating/updating-restricted-network-cluster.html - This includes updating the local private registry with the updated mirror images that you want to upgrade to. Actual results: Upon performing the 'oc adm upgrade ...' command, the clusterversion output will show the status of the cluster is working toward the upgrade - downloading update. But it will eventually fail with the following error: # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.14 True True 3h6m Unable to apply 4.6.0-rc.2: the workload openshift-etcd-operator/etcd-operator has not yet successfully rolled out It will go back and forth between downloading and failing with this error indefinitely. We have left this up for hours and up to a day, but it will just continually loop. Expected results: Upgrade current OCP cluster on restricted network should succeed. Additional info: Performing an upgrade on a regular connected cluster to 4.6 rc will succeed. Is a disconnected install cluster upgrade using a connected cluster(non-restricted) procedure supported? We tested additional install options, here is the matrix of those results: 1. 4.5.13 connected install, to 4.6.0-rc.1 and rc.2 connected upgrade succeeds. 2. 4.5.14 connected install, to 4.6.0-rc.1 and rc.2 connected upgrade succeeds. 3. 4.5.14 disconnected install, to 4.6.0-rc.1 and rc.2 connected upgrade fails 4. 4.5.13 disconnected install, to 4.6.0-rc.1 and rc.2 disconnected upgrade fails 5. 4.5.14 disconnected install, to 4.6.0-rc.1 and rc.2 disconnected upgrade fails We have also performed the same disconnected upgrade tests on z/KVM as a comparison, and we are unable to successful upgrade there as well. Please let me know if you need additional info. Thank you, -Phil Chan
Phil, Could you please collect the must-gather logs and provide them so we can investigate ? Thanks, Prashanth
Hi Phil, in addition to Prashanth's request in Comment 1, please also kindly review the below slide for a list of required files to attach: https://docs.google.com/presentation/d/1KMWaIAMHn_0kdt38dV2WeZTta57cdo8EJwlQutux6KE/edit#slide=id.g5391221dc4_0_68
oc describe clusteroperator -o yaml might help (in addition to oc adm must-gather)
oc describe clusteroperator etcd -o yaml - sorry, missed etcd
After successfully installing OCP 4.5.15 using the disconnected install method, and then attempting a connected upgrade to 4.6.0-rc.4, the upgrade hangs and infinitely alternates displaying the following status between these 2 messages from the "oc get clusterversion command": 1. Working towards 4.6.0-rc.4: 11% complete 2. Unable to apply 4.6.0-rc.4: the cluster operator etcd has not yet successfully rolled out When attempting to gather "oc adm must-gather" fails with this information: [root@OSPAMGR2 ~]# oc adm must-gather [must-gather ] OUT unable to resolve the imagestream tag openshift/must-gather:latest [must-gather ] OUT [must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest [must-gather ] OUT namespace/openshift-must-gather-rb6jk created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-59wpl created [must-gather ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created [must-gather-g47w4] POD standard_init_linux.go:210: exec user process caused "exec format error" [must-gather-g47w4] OUT waiting for gather to complete [must-gather-g47w4] OUT gather never finished: openshift-must-gather-rb6jk/must-gather-g47w4 unexpectedly terminated: exit code: 1, reason: Error, message: [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-59wpl deleted [must-gather ] OUT namespace/openshift-must-gather-rb6jk deleted error: gather never finished for pod must-gather-g47w4: openshift-must-gather-rb6jk/must-gather-g47w4 unexpectedly terminated: exit code: 1, reason: Error, message:
[root@OSPAMGR2 4.6.0-rc.2]# oc describe clusteroperator etcd Name: etcd Namespace: Labels: <none> Annotations: exclude.release.openshift.io/internal-openshift-hosted: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-10-14T15:32:36Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:exclude.release.openshift.io/internal-openshift-hosted: f:spec: f:status: .: f:extension: f:relatedObjects: Manager: cluster-version-operator Operation: Update Time: 2020-10-14T15:32:36Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:versions: Manager: cluster-etcd-operator Operation: Update Time: 2020-10-14T15:41:08Z Resource Version: 14411 Self Link: /apis/config.openshift.io/v1/clusteroperators/etcd UID: c874bb13-2b55-4276-8d11-f581c8913cf0 Spec: Status: Conditions: Last Transition Time: 2020-10-14T15:36:38Z Message: NodeControllerDegraded: All master nodes are ready EtcdMembersDegraded: No unhealthy members found Reason: AsExpected Status: False Type: Degraded Last Transition Time: 2020-10-14T15:41:08Z Message: NodeInstallerProgressing: 3 nodes are at revision 3 EtcdMembersProgressing: No unstarted etcd members found Reason: AsExpected Status: False Type: Progressing Last Transition Time: 2020-10-14T15:35:48Z Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 3 EtcdMembersAvailable: 3 members are available Reason: AsExpected Status: True Type: Available Last Transition Time: 2020-10-14T15:34:02Z Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: etcds Group: Name: openshift-config Resource: namespaces Group: Name: openshift-config-managed Resource: namespaces Group: Name: openshift-etcd-operator Resource: namespaces Group: Name: openshift-etcd Resource: namespaces Versions: Name: operator Version: 4.5.15 Name: raw-internal Version: 4.5.15 Name: etcd Version: 4.5.15 Events: <none> [root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 4.6.0-rc.2]# oc get pods -n openshift-etcd NAME READY STATUS RESTARTS AGE etcd-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m etcd-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 151m etcd-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m installer-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m installer-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m installer-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m installer-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m revision-pruner-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m revision-pruner-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m revision-pruner-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m revision-pruner-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m [root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 4.6.0-rc.2]# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.15 True False False 150m cloud-credential 4.5.15 True False False 162m cluster-autoscaler 4.5.15 True False False 154m config-operator 4.5.15 True False False 155m console 4.5.15 True False False 146m csi-snapshot-controller 4.5.15 True False False 156m dns 4.5.15 True False False 160m etcd 4.5.15 True False False 159m image-registry 4.5.15 True False False 155m ingress 4.5.15 True False False 155m insights 4.5.15 True False False 155m kube-apiserver 4.5.15 True False False 159m kube-controller-manager 4.5.15 True False False 159m kube-scheduler 4.5.15 True False False 158m kube-storage-version-migrator 4.5.15 True False False 155m machine-api 4.5.15 True False False 155m machine-approver 4.5.15 True False False 157m machine-config 4.5.15 True False False 159m marketplace 4.5.15 True False False 155m monitoring 4.5.15 True False False 149m network 4.5.15 True False False 161m node-tuning 4.5.15 True False False 161m openshift-apiserver 4.5.15 True False False 156m openshift-controller-manager 4.5.15 True False False 156m openshift-samples 4.5.15 True True True 154m operator-lifecycle-manager 4.5.15 True False False 160m operator-lifecycle-manager-catalog 4.5.15 True False False 160m operator-lifecycle-manager-packageserver 4.5.15 True False False 156m service-ca 4.5.15 True False False 161m storage 4.5.15 True False False 156m [root@OSPAMGR2 4.6.0-rc.2]# oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE openshift-apiserver-operator openshift-apiserver-operator-ddc649644-6bdkt 1/1 Running 2 162m openshift-apiserver apiserver-65b4c9ffd6-5bpqj 1/1 Running 0 159m openshift-apiserver apiserver-65b4c9ffd6-prlwz 1/1 Running 0 159m openshift-apiserver apiserver-65b4c9ffd6-xqqzb 1/1 Running 0 159m openshift-authentication-operator authentication-operator-669844ffdc-bqc6s 1/1 Running 2 162m openshift-authentication oauth-openshift-8c8c8bb8f-bdngx 1/1 Running 0 155m openshift-authentication oauth-openshift-8c8c8bb8f-l8lz9 1/1 Running 0 155m openshift-cloud-credential-operator cloud-credential-operator-c7879d5dd-tppq2 1/1 Running 0 156m openshift-cluster-machine-approver machine-approver-588d44976d-4v55h 2/2 Running 0 162m openshift-cluster-node-tuning-operator cluster-node-tuning-operator-64fd77f697-8tl6z 1/1 Running 0 162m openshift-cluster-node-tuning-operator tuned-6cdwf 1/1 Running 0 161m openshift-cluster-node-tuning-operator tuned-g76gl 1/1 Running 0 161m openshift-cluster-node-tuning-operator tuned-krg4p 1/1 Running 0 156m openshift-cluster-node-tuning-operator tuned-mchv6 1/1 Running 0 161m openshift-cluster-node-tuning-operator tuned-mzqcg 1/1 Running 0 156m openshift-cluster-samples-operator cluster-samples-operator-5d68b5695c-nnl75 2/2 Running 0 155m openshift-cluster-storage-operator cluster-storage-operator-b7f6c797c-cpgdj 1/1 Running 0 156m openshift-cluster-storage-operator csi-snapshot-controller-864c4556cb-5tg6b 1/1 Running 0 156m openshift-cluster-storage-operator csi-snapshot-controller-operator-566775d468-zgbmr 1/1 Running 1 162m openshift-cluster-version cluster-version-operator-96c8b8d88-8r2qc 1/1 Running 0 37m openshift-cluster-version version--bxtts-dpjl8 0/1 Completed 0 38m openshift-config-operator openshift-config-operator-9d4cf66db-zr6tk 1/1 Running 0 155m openshift-console-operator console-operator-5df8c8d4b5-6dz8h 1/1 Running 0 156m openshift-console console-595cc745ff-2pndb 1/1 Running 0 147m openshift-console console-595cc745ff-g98xl 1/1 Running 0 147m openshift-console downloads-547fd9d88d-fwnbr 1/1 Running 0 156m openshift-console downloads-547fd9d88d-ljp7p 1/1 Running 0 156m openshift-controller-manager-operator openshift-controller-manager-operator-54f65787d7-rskbz 1/1 Running 2 162m openshift-controller-manager controller-manager-2sb6h 1/1 Running 0 156m openshift-controller-manager controller-manager-dqsj7 1/1 Running 0 156m openshift-controller-manager controller-manager-tzf2z 1/1 Running 0 156m openshift-dns-operator dns-operator-546f67b8d-9cbfv 2/2 Running 0 162m openshift-dns dns-default-78pjf 3/3 Running 0 160m openshift-dns dns-default-94n69 3/3 Running 0 156m openshift-dns dns-default-qp5q5 3/3 Running 0 160m openshift-dns dns-default-wn9d2 3/3 Running 0 160m openshift-dns dns-default-xc9gr 3/3 Running 0 156m openshift-etcd-operator etcd-operator-7cb94c56-4fjdp 0/1 ImagePullBackOff 0 37m openshift-etcd etcd-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 155m openshift-etcd etcd-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 154m openshift-etcd etcd-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 154m openshift-etcd installer-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-etcd installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-etcd installer-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-etcd installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-etcd installer-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-etcd installer-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-etcd revision-pruner-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-etcd revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-etcd revision-pruner-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-etcd revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-etcd revision-pruner-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-etcd revision-pruner-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-image-registry cluster-image-registry-operator-7d9bd9765f-hj4hv 2/2 Running 0 156m openshift-image-registry node-ca-5lzsx 1/1 Running 0 155m openshift-image-registry node-ca-qmsn4 1/1 Running 0 155m openshift-image-registry node-ca-qpqlr 1/1 Running 0 155m openshift-image-registry node-ca-tzghs 1/1 Running 0 155m openshift-image-registry node-ca-wx7wz 1/1 Running 0 155m openshift-ingress-operator ingress-operator-b4948b49c-x47q7 2/2 Running 0 156m openshift-ingress router-default-99598fc9c-6cb55 1/1 Running 0 155m openshift-ingress router-default-99598fc9c-zxfgn 1/1 Running 0 155m openshift-insights insights-operator-859b6f7588-k24qs 1/1 Running 0 156m openshift-kube-apiserver-operator kube-apiserver-operator-5869c75df6-d5pj6 1/1 Running 2 162m openshift-kube-apiserver installer-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-apiserver installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-apiserver installer-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-apiserver installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-apiserver installer-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-apiserver installer-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver installer-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver installer-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 148m openshift-kube-apiserver installer-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 150m openshift-kube-apiserver installer-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-apiserver kube-apiserver-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 148m openshift-kube-apiserver kube-apiserver-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 150m openshift-kube-apiserver kube-apiserver-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m openshift-kube-apiserver revision-pruner-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-apiserver revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-apiserver revision-pruner-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-apiserver revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-apiserver revision-pruner-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver revision-pruner-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver revision-pruner-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-apiserver revision-pruner-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 147m openshift-kube-apiserver revision-pruner-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 149m openshift-kube-apiserver revision-pruner-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager-operator kube-controller-manager-operator-9c446d7cc-2q5t7 1/1 Running 2 162m openshift-kube-controller-manager installer-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-controller-manager installer-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager installer-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager installer-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-controller-manager installer-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-controller-manager installer-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-controller-manager installer-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-controller-manager installer-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-controller-manager installer-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager installer-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-controller-manager installer-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-controller-manager kube-controller-manager-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 151m openshift-kube-controller-manager kube-controller-manager-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m openshift-kube-controller-manager kube-controller-manager-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m openshift-kube-controller-manager revision-pruner-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager revision-pruner-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-controller-manager revision-pruner-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager revision-pruner-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m openshift-kube-controller-manager revision-pruner-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-controller-manager revision-pruner-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-controller-manager revision-pruner-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-controller-manager revision-pruner-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-controller-manager revision-pruner-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager revision-pruner-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager revision-pruner-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-scheduler-operator openshift-kube-scheduler-operator-6674f9c8fd-h87fm 1/1 Running 2 162m openshift-kube-scheduler installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 161m openshift-kube-scheduler installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-scheduler installer-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler installer-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-scheduler installer-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-scheduler installer-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler installer-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler installer-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-scheduler installer-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 150m openshift-kube-scheduler installer-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-scheduler installer-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-scheduler openshift-kube-scheduler-master-0.pok-25.pok.stglabs.ibm.com 2/2 Running 0 150m openshift-kube-scheduler openshift-kube-scheduler-master-1.pok-25.pok.stglabs.ibm.com 2/2 Running 0 153m openshift-kube-scheduler openshift-kube-scheduler-master-2.pok-25.pok.stglabs.ibm.com 2/2 Running 0 151m openshift-kube-scheduler revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-scheduler revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-scheduler revision-pruner-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler revision-pruner-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m openshift-kube-scheduler revision-pruner-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-scheduler revision-pruner-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler revision-pruner-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-scheduler revision-pruner-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-scheduler revision-pruner-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 149m openshift-kube-scheduler revision-pruner-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-scheduler revision-pruner-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 150m openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-64d5f49dfc-2zjgl 1/1 Running 2 162m openshift-kube-storage-version-migrator migrator-7f6f887d6c-p58rs 1/1 Running 0 161m openshift-machine-api cluster-autoscaler-operator-845984ffc9-nqf6z 2/2 Running 0 155m openshift-machine-api machine-api-operator-6f8dc9b664-p7hsz 2/2 Running 0 156m openshift-machine-config-operator etcd-quorum-guard-669f5cc5cc-84tlp 1/1 Running 0 160m openshift-machine-config-operator etcd-quorum-guard-669f5cc5cc-mn2h9 1/1 Running 0 160m openshift-machine-config-operator etcd-quorum-guard-669f5cc5cc-r26h4 1/1 Running 0 160m openshift-machine-config-operator machine-config-controller-5cd67cc8fb-6l5sv 1/1 Running 1 161m openshift-machine-config-operator machine-config-daemon-577r5 2/2 Running 0 156m openshift-machine-config-operator machine-config-daemon-77rvm 2/2 Running 0 161m openshift-machine-config-operator machine-config-daemon-j568n 2/2 Running 0 161m openshift-machine-config-operator machine-config-daemon-pfq7w 2/2 Running 0 161m openshift-machine-config-operator machine-config-daemon-zxhp6 2/2 Running 0 156m openshift-machine-config-operator machine-config-operator-68887c4cdf-cjxmd 1/1 Running 1 162m openshift-machine-config-operator machine-config-server-657hv 1/1 Running 0 160m openshift-machine-config-operator machine-config-server-cdm69 1/1 Running 0 160m openshift-machine-config-operator machine-config-server-tjhc5 1/1 Running 0 160m openshift-marketplace certified-operators-69cdf66448-jsz8g 1/1 Running 0 94m openshift-marketplace community-operators-cc799f499-xfgbc 1/1 Running 0 155m openshift-marketplace marketplace-operator-668d865746-7whkk 1/1 Running 0 156m openshift-marketplace redhat-marketplace-d9c7bc85b-fpqsr 1/1 Running 0 155m openshift-marketplace redhat-operators-95f6487c8-qchh2 1/1 Running 0 155m openshift-monitoring alertmanager-main-0 5/5 Running 0 155m openshift-monitoring alertmanager-main-1 5/5 Running 0 155m openshift-monitoring alertmanager-main-2 5/5 Running 0 155m openshift-monitoring cluster-monitoring-operator-55966bbf46-j2nnl 2/2 Running 2 156m openshift-monitoring grafana-67b9fcf78-srgst 2/2 Running 0 155m openshift-monitoring kube-state-metrics-746ddc9785-p4pwc 3/3 Running 0 155m openshift-monitoring node-exporter-jxpqq 2/2 Running 0 155m openshift-monitoring node-exporter-n4gdk 2/2 Running 0 155m openshift-monitoring node-exporter-nsplw 2/2 Running 0 155m openshift-monitoring node-exporter-sjjv5 2/2 Running 0 155m openshift-monitoring node-exporter-tfb7h 2/2 Running 0 155m openshift-monitoring openshift-state-metrics-77cc7ff79f-gsq9c 3/3 Running 0 155m openshift-monitoring prometheus-adapter-85d69fbf6f-b57cv 1/1 Running 0 149m openshift-monitoring prometheus-adapter-85d69fbf6f-xkj69 1/1 Running 0 149m openshift-monitoring prometheus-k8s-0 7/7 Running 1 149m openshift-monitoring prometheus-k8s-1 7/7 Running 1 149m openshift-monitoring prometheus-operator-59bcf8b984-5wc74 2/2 Running 0 155m openshift-monitoring thanos-querier-84455df89d-mrzj8 4/4 Running 0 149m openshift-monitoring thanos-querier-84455df89d-trf7p 4/4 Running 0 149m openshift-multus multus-299nz 1/1 Running 0 162m openshift-multus multus-8wjk6 1/1 Running 0 162m openshift-multus multus-8wlc8 1/1 Running 0 156m openshift-multus multus-admission-controller-bc24q 2/2 Running 0 161m openshift-multus multus-admission-controller-p2tlb 2/2 Running 0 161m openshift-multus multus-admission-controller-v588v 2/2 Running 0 161m openshift-multus multus-fkg5j 1/1 Running 0 162m openshift-multus multus-gngft 1/1 Running 0 156m openshift-network-operator network-operator-6f6657fddb-88nhf 1/1 Running 0 162m openshift-operator-lifecycle-manager catalog-operator-6dd5f7776-gtxjw 1/1 Running 0 162m openshift-operator-lifecycle-manager olm-operator-559cfb8bb6-8gps7 1/1 Running 0 162m openshift-operator-lifecycle-manager packageserver-8d6d667b8-99pdj 1/1 Running 0 160m openshift-operator-lifecycle-manager packageserver-8d6d667b8-p5lbv 1/1 Running 0 160m openshift-sdn ovs-2wkf7 1/1 Running 0 156m openshift-sdn ovs-8857l 1/1 Running 0 156m openshift-sdn ovs-8d594 1/1 Running 0 161m openshift-sdn ovs-9fflj 1/1 Running 0 161m openshift-sdn ovs-jpcqp 1/1 Running 0 161m openshift-sdn sdn-5vs7l 1/1 Running 0 161m openshift-sdn sdn-controller-94qx8 1/1 Running 0 161m openshift-sdn sdn-controller-dbjb2 1/1 Running 0 161m openshift-sdn sdn-controller-jj5fl 1/1 Running 1 161m openshift-sdn sdn-cwpqc 1/1 Running 0 161m openshift-sdn sdn-hkq6v 1/1 Running 0 156m openshift-sdn sdn-nt4wn 1/1 Running 0 161m openshift-sdn sdn-sjl48 1/1 Running 0 156m openshift-service-ca-operator service-ca-operator-66f96555b-wlb29 1/1 Running 2 162m openshift-service-ca service-ca-75b767bcc4-j7wng 1/1 Running 1 161m openshift-service-catalog-removed openshift-service-catalog-apiserver-remover-9665h 0/1 Completed 0 149m openshift-service-catalog-removed openshift-service-catalog-controller-manager-remover-mn6wv 0/1 Completed 0 149m [root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 4.6.0-rc.2]# oc describe pod etcd-operator-7cb94c56-4fjdp -n openshift-etcd-operator Name: etcd-operator-7cb94c56-4fjdp Namespace: openshift-etcd-operator Priority: 2000000000 Priority Class Name: system-cluster-critical Node: master-2.pok-25.pok.stglabs.ibm.com/10.20.116.13 Start Time: Wed, 14 Oct 2020 13:38:23 -0400 Labels: app=etcd-operator pod-template-hash=7cb94c56 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.0.49" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.0.49" ], "default": true, "dns": {} }] Status: Pending IP: 10.130.0.49 IPs: IP: 10.130.0.49 Controlled By: ReplicaSet/etcd-operator-7cb94c56 Containers: etcd-operator: Container ID: Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 Image ID: Port: 8443/TCP Host Port: 0/TCP Command: cluster-etcd-operator operator Args: --config=/var/run/configmaps/config/config.yaml -v=4 --terminate-on-files=/var/run/secrets/serving-cert/tls.crt --terminate-on-files=/var/run/secrets/serving-cert/tls.key State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Requests: cpu: 10m memory: 50Mi Environment: IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25ce82e8c5117ccfc7ca6edd84faeadfd72a4298dcb78bfc964e3f633f65837e OPERATOR_IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 OPERATOR_IMAGE_VERSION: 4.6.0-rc.4 OPERAND_IMAGE_VERSION: 4.6.0-rc.4 POD_NAME: etcd-operator-7cb94c56-4fjdp (v1:metadata.name) Mounts: /var/run/configmaps/config from config (rw) /var/run/configmaps/etcd-ca from etcd-ca (rw) /var/run/configmaps/etcd-service-ca from etcd-service-ca (rw) /var/run/secrets/etcd-client from etcd-client (rw) /var/run/secrets/kubernetes.io/serviceaccount from etcd-operator-token-qlxwf (ro) /var/run/secrets/serving-cert from serving-cert (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: serving-cert: Type: Secret (a volume populated by a Secret) SecretName: etcd-operator-serving-cert Optional: true config: Type: ConfigMap (a volume populated by a ConfigMap) Name: etcd-operator-config Optional: false etcd-ca: Type: ConfigMap (a volume populated by a ConfigMap) Name: etcd-ca-bundle Optional: false etcd-service-ca: Type: ConfigMap (a volume populated by a ConfigMap) Name: etcd-service-ca-bundle Optional: false etcd-client: Type: Secret (a volume populated by a Secret) SecretName: etcd-client Optional: false etcd-operator-token-qlxwf: Type: Secret (a volume populated by a Secret) SecretName: etcd-operator-token-qlxwf Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 120s node.kubernetes.io/unreachable:NoExecute for 120s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 50m default-scheduler Successfully assigned openshift-etcd-operator/etcd-operator-7cb94c56-4fjdp to master-2.pok-25.pok.stglabs.ibm.com Normal AddedInterface 50m multus Add eth0 [10.130.0.49/23] Normal Pulling 49m (x4 over 50m) kubelet Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32" Warning Failed 49m (x4 over 50m) kubelet Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32": rpc error: code = Unknown desc = (Mirrors also failed: [bastion:5000/ocp4/openshift4@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32: Error reading manifest sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 in bastion:5000/ocp4/openshift4: manifest unknown: manifest unknown]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32: Error reading manifest sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized Warning Failed 49m (x4 over 50m) kubelet Error: ErrImagePull Normal BackOff 15m (x152 over 50m) kubelet Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32" Warning Failed 32s (x217 over 50m) kubelet Error: ImagePullBackOff [root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 4.6.0-rc.2]# oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32: unauthorized: access to the requested resource is not authorized [root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 /]# oc image info --registry-config='/root/disconnectedinstall/pull-secret.json' quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 Name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 Media Type: application/vnd.docker.distribution.manifest.v2+json Created: 7d ago Image Size: 102.2MB in 5 layers Layers: 71.38MB sha256:024e289f9d3059db8f47e89e0e13314df01f8651dcf9afb50c4c68d58cb92520 1.79kB sha256:34fd41f9801e60f1598818f1a245f68828bf63da35256d5d904a8880d50061aa 4.474MB sha256:cc41e5e3f4a3310408d5cba6603c88bbbc3169dda98ae8ec995044498f9e28ea 10.92MB sha256:a8eac247d57c251bd1da15d4eb49e0589efaf39584419a69348e3aee3ec63d70 15.41MB sha256:c153ba408d51a3f4f4bb1edb66a14cb0e8a4a437d601a153f7d2e45852a1e6be OS: linux Arch: s390x Command: /bin/bash User: 0 Environment: __doozer=merge BUILD_RELEASE=202010062159.p0 BUILD_VERSION=v4.6.0 OS_GIT_MAJOR=4 OS_GIT_MINOR=6 OS_GIT_PATCH=0 OS_GIT_TREE_STATE=clean OS_GIT_VERSION=4.6.0-202010062159.p0-3c9d46e SOURCE_GIT_TREE_STATE=clean OS_GIT_COMMIT=3c9d46e SOURCE_DATE_EPOCH=1601490126 SOURCE_GIT_COMMIT=3c9d46eb6987428dd8ccb65802606eb204737495 SOURCE_GIT_TAG=v0.0.0-alpha.0-572-g3c9d46eb SOURCE_GIT_URL=https://github.com/openshift/cluster-etcd-operator GODEBUG=x509ignoreCN=0 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci Labels: License=GPLv2+ architecture=s390x build-date=2020-10-06T22:09:54.796336 com.redhat.build-host=s390-c1-vm-03.prod.osbs.eng.bos.redhat.com com.redhat.component=cluster-etcd-operator-container com.redhat.license_terms=https://www.redhat.com/agreements description=This is the base image from which all OpenShift Container Platform images inherit. distribution-scope=public io.k8s.description=This is the base image from which all OpenShift Container Platform images inherit. io.k8s.display-name=OpenShift Container Platform RHEL 7 Base io.openshift.build.commit.id=3c9d46eb6987428dd8ccb65802606eb204737495 io.openshift.build.commit.url=https://github.com/openshift/cluster-etcd-operator/commit/3c9d46eb6987428dd8ccb65802606eb204737495 io.openshift.build.source-location=https://github.com/openshift/cluster-etcd-operator io.openshift.expose-services= io.openshift.maintainer.component=Etcd Operator io.openshift.maintainer.product=OpenShift Container Platform io.openshift.release.operator=true io.openshift.tags=openshift,base maintainer=Red Hat, Inc. name=openshift/ose-cluster-etcd-operator release=202010062159.p0 summary=Provides the latest release of Red Hat Universal Base Image 8. url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-cluster-etcd-operator/images/v4.6.0-202010062159.p0 vcs-ref=dd84cdd61cb63a842385f1a00502b032827e297e vcs-type=git vendor=Red Hat, Inc. version=v4.6.0 [root@OSPAMGR2 /]#
Created attachment 1721557 [details] master-2 kubelet log
oc adm upgrade --force --allow-explicit-upgrade --to-image $target_image
Jeremy and Prashanth, Thank you for all your assistance (including with the above posts) with the debug of the disconnected install to connected upgrade for helping us further debug the disconnected install to disconnected upgrade issue. Dynamically changing the cluster pull-secret from disconnected install specific (local mirror registry) to connected install/upgrade specific (quay.io) resolved the issue. Specifically, using the command: oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=authfile where authfile is the updated connected install specific (quay.io) pull-secret file. Thank you, Kyle
Thank you Jeremy and Prashanth for your help in debugging our disconnected install to disconnected upgrade issue. As we determined in our meeting, there was a typo on the 'oc adm upgrade' command that we issued where we did not include an "@" before the <sha256_sum_value>. Since we were issuing the upgrade with force, it accepted the command overriding any verifications. The correct upgrade command that we issued on our test cluster from OCP 4.6 (4.6.0-0.nightly-s390x-2020-10-02-202613) to 4.6 (4.6.0-0.nightly-s390x-2020-10-08-182421) is the following: # oc adm upgrade --force --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}@sha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1 We may need to highlight the importance of the @ sign in the documentation? After issuing the correct upgrade command, the "working towards" now shows the percentage complete and is progressing. We are going to fully test this and run several 4.5 to 4.6 (4.6.0-rc.3) upgrades on z/VM and z/KVM. We will update the results here when complete. Thank you, -Phil
Jeremy and Prashanth, Thank you again for all your assistance with this issue. Here's an update on the current zVM hosted OCP 4.6 disconnected install upgrade testing. 1. A zVM based OCP 4.6 disconnected install and upgrade from 4.5 (4.5.12) to 4.6 (4.6.0-rc.3) was successful. 2. Additional zVM hosted OCP 4.6 disconnected install upgrade tests are in progress. These tests are using OCP 4.5.14 and 4.5.15 as 4.5 disconnected install bases, upgrading to OCP 4.6.0-rc.3 and 4.6.0-rc.4. We will be following up here with the results of these and other additional OCP 4.6 disconnected install and upgrade tests results for zKVM and zVM hosted clusters later today. Thank you, Kyle
Jeremy and Prashanth, Just a quick reminder to open a new bugzilla (or we can, whichever is your preference) for the OCP 4.5 disconnected install "oc adm must-gather" issue where when running this command on a OCP 4.5 disconnected installed cluster it fails, as opposed to a OCP 4.5 connected installed cluster where the "oc adm must-gather" command succeeds. 1. Please see comment #5 above, specifically the error message from the "oc adm must-gather" command which appears to be an x86 module loaded instead of an s390x module: standard_init_linux.go:210: exec user process caused "exec format error" 2. We have tested further and have seen this same issue on OCP 4.5.12, 4.5.13, 4.5.14, and 4.5.15 for disconnected installs only, and not with connected installs. Thank you, Kyle
Kyle/Phil, Thanks! If it is ok with you , I am going to close this bug. Could you also open a new BZ for them must gather issue and provide this information in that BZ: - run an `oc image info <must-gather-image>` where the must-gather-image is the mirrored must gather image. This will give us an idea of what got mirrored. Thanks Prashanth
I've filed the new issue here: https://bugzilla.redhat.com/show_bug.cgi?id=1888738
(In reply to Philip Chan from comment #15) > Thank you Jeremy and Prashanth for your help in debugging our disconnected > install to disconnected upgrade issue. As we determined in our meeting, > there was a typo on the 'oc adm upgrade' command that we issued where we did > not include an "@" before the <sha256_sum_value>. Can you paste the command you ran with the incorrect pullspec? I'd expect us to do a better job complaining about pullspecs that lacked a digest (e.g. via [1]). [1]: https://github.com/openshift/oc/pull/390
Hi, The incorrect command that was issued when attempting to upgrade to 4.6.0-0.nightly-s390x-2020-10-08-18242 was the following: # oc adm upgrade --force --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}sha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1 Thanks, Phil
Checking with a recent-ish client: $ oc version --client Client Version: 4.5.11 $ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-releasesha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1 error: --to-image must be a by-digest pull spec, unless --force is also set, because release images that are not accessed via digest cannot be verified by the cluster. Even when --force is set, using tags is not recommended, although we continue to allow it for backwards compatibility So that's good. But you point out that you used --force, the by-tag pullspec becomes a non-fatal stderr warning: $ oc adm upgrade --force --to-image quay.io/openshift-release-dev/ocp-releasesha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1 warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead Error from server (Forbidden): clusterversions.config.openshift.io "version" is forbidden: User "wking" cannot get clusterversions.config.openshift.io at the cluster scope: no RBAC policy matched The failure there is because I'm not an admin on the cluster I'm authenticating against, but it shows the logged pullspec warning. Also not clear to me why you are using --force. Are you not mirroring the signature as a ConfigMap [1]? Official docs around this in [2] need some polish still, but the procedure in [1] is what you want for sneaker-net mirroring. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1837675#c5 [2]: https://docs.openshift.com/container-platform/4.5/updating/updating-restricted-network-cluster.html#updating-restricted-network-image-signature-configmap
Jeremy and Prashanth, The following zVM hosted OCP 4.6 cluster disconnected install builds were successfully upgraded with the corresponding disconnected upgrade builds. Disconnected install build Disconnected upgrade build ========================== ========================== 1. 4.5.4 4.6.0-rc.3 2. 4.5.4 4.6.0-rc.4 3. 4.5.5 4.6.0-rc.3 4. 4.5.5 4.6.0-rc.4 5. 4.5.12 4.6.0-rc.3 6. 4.5.12 4.6.0-rc.4 7. 4.5.14 4.6.0-rc.3 8. 4.5.14 4.6.0-rc.4 9. 4.5.15 4.6.0-rc.3 10. 4.5.15 4.6.0-rc.4 Thank you, Kyle
Hi Trevor, That is correct, we do not need to use the --force for the upgrade. The --force was left as a result of our debug (4.6.x -> 4.6.x). I just ran a quick upgrade test without the --force and that succeeded: # oc adm upgrade --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}@sha256:2d286c599fd767ea5041c9e0b11863efa321daaf9616d640199601e8dd9da3d4 warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway Updating to release image bastion:5000/ocp4/openshift4@sha256:2d286c599fd767ea5041c9e0b11863efa321daaf9616d640199601e8dd9da3d4 With regards to the ConfigMap, I agree that we need some work with the official documentation. I passed in the --apply-release-image-signature option when performing the mirroring, as documented, if that is done, we did not perform the ConfigMap steps. Here's what we passed: # oc adm -a ${LOCAL_SECRET_JSON} release mirror --from=quay.io/openshift-release-dev/ocp-release@sha256:2d286c599fd767ea5041c9e0b11863efa321daaf9616d640199601e8dd9da3d4 --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE} --apply-release-image-signature Hope that info helps, please let me know if you need any additional info. Thanks, Phil