Bug 1888065
| Summary: | Upgrade to 4.6 disconnected cluster will fail to complete | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Philip Chan <chanphil> | ||||
| Component: | Multi-Arch | Assignee: | Prashanth Sundararaman <psundara> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Jeremy Poulin <jpoulin> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.6 | CC: | christian.lapolt, danili, Holger.Wolf, krmoser, mifiedle, nbziouec, psundara, wking | ||||
| Target Milestone: | --- | Keywords: | Upgrades | ||||
| Target Release: | 4.6.0 | Flags: | chanphil:
needinfo+
|
||||
| Hardware: | s390x | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-10-15 17:55:38 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1881153 | ||||||
| Attachments: |
|
||||||
|
Description
Philip Chan
2020-10-13 22:28:23 UTC
Phil, Could you please collect the must-gather logs and provide them so we can investigate ? Thanks, Prashanth Hi Phil, in addition to Prashanth's request in Comment 1, please also kindly review the below slide for a list of required files to attach: https://docs.google.com/presentation/d/1KMWaIAMHn_0kdt38dV2WeZTta57cdo8EJwlQutux6KE/edit#slide=id.g5391221dc4_0_68 oc describe clusteroperator -o yaml might help (in addition to oc adm must-gather) oc describe clusteroperator etcd -o yaml - sorry, missed etcd After successfully installing OCP 4.5.15 using the disconnected install method, and then attempting a connected upgrade to 4.6.0-rc.4, the upgrade hangs and infinitely alternates displaying the following status between these 2 messages from the "oc get clusterversion command": 1. Working towards 4.6.0-rc.4: 11% complete 2. Unable to apply 4.6.0-rc.4: the cluster operator etcd has not yet successfully rolled out When attempting to gather "oc adm must-gather" fails with this information: [root@OSPAMGR2 ~]# oc adm must-gather [must-gather ] OUT unable to resolve the imagestream tag openshift/must-gather:latest [must-gather ] OUT [must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest [must-gather ] OUT namespace/openshift-must-gather-rb6jk created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-59wpl created [must-gather ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created [must-gather-g47w4] POD standard_init_linux.go:210: exec user process caused "exec format error" [must-gather-g47w4] OUT waiting for gather to complete [must-gather-g47w4] OUT gather never finished: openshift-must-gather-rb6jk/must-gather-g47w4 unexpectedly terminated: exit code: 1, reason: Error, message: [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-59wpl deleted [must-gather ] OUT namespace/openshift-must-gather-rb6jk deleted error: gather never finished for pod must-gather-g47w4: openshift-must-gather-rb6jk/must-gather-g47w4 unexpectedly terminated: exit code: 1, reason: Error, message:
[root@OSPAMGR2 4.6.0-rc.2]# oc describe clusteroperator etcd
Name: etcd
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-10-14T15:32:36Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:spec:
f:status:
.:
f:extension:
f:relatedObjects:
Manager: cluster-version-operator
Operation: Update
Time: 2020-10-14T15:32:36Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:versions:
Manager: cluster-etcd-operator
Operation: Update
Time: 2020-10-14T15:41:08Z
Resource Version: 14411
Self Link: /apis/config.openshift.io/v1/clusteroperators/etcd
UID: c874bb13-2b55-4276-8d11-f581c8913cf0
Spec:
Status:
Conditions:
Last Transition Time: 2020-10-14T15:36:38Z
Message: NodeControllerDegraded: All master nodes are ready
EtcdMembersDegraded: No unhealthy members found
Reason: AsExpected
Status: False
Type: Degraded
Last Transition Time: 2020-10-14T15:41:08Z
Message: NodeInstallerProgressing: 3 nodes are at revision 3
EtcdMembersProgressing: No unstarted etcd members found
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2020-10-14T15:35:48Z
Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 3
EtcdMembersAvailable: 3 members are available
Reason: AsExpected
Status: True
Type: Available
Last Transition Time: 2020-10-14T15:34:02Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: etcds
Group:
Name: openshift-config
Resource: namespaces
Group:
Name: openshift-config-managed
Resource: namespaces
Group:
Name: openshift-etcd-operator
Resource: namespaces
Group:
Name: openshift-etcd
Resource: namespaces
Versions:
Name: operator
Version: 4.5.15
Name: raw-internal
Version: 4.5.15
Name: etcd
Version: 4.5.15
Events: <none>
[root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 4.6.0-rc.2]# oc get pods -n openshift-etcd NAME READY STATUS RESTARTS AGE etcd-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m etcd-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 151m etcd-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m installer-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m installer-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m installer-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m installer-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m revision-pruner-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m revision-pruner-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m revision-pruner-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m revision-pruner-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m [root@OSPAMGR2 4.6.0-rc.2]# [root@OSPAMGR2 4.6.0-rc.2]# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.15 True False False 150m cloud-credential 4.5.15 True False False 162m cluster-autoscaler 4.5.15 True False False 154m config-operator 4.5.15 True False False 155m console 4.5.15 True False False 146m csi-snapshot-controller 4.5.15 True False False 156m dns 4.5.15 True False False 160m etcd 4.5.15 True False False 159m image-registry 4.5.15 True False False 155m ingress 4.5.15 True False False 155m insights 4.5.15 True False False 155m kube-apiserver 4.5.15 True False False 159m kube-controller-manager 4.5.15 True False False 159m kube-scheduler 4.5.15 True False False 158m kube-storage-version-migrator 4.5.15 True False False 155m machine-api 4.5.15 True False False 155m machine-approver 4.5.15 True False False 157m machine-config 4.5.15 True False False 159m marketplace 4.5.15 True False False 155m monitoring 4.5.15 True False False 149m network 4.5.15 True False False 161m node-tuning 4.5.15 True False False 161m openshift-apiserver 4.5.15 True False False 156m openshift-controller-manager 4.5.15 True False False 156m openshift-samples 4.5.15 True True True 154m operator-lifecycle-manager 4.5.15 True False False 160m operator-lifecycle-manager-catalog 4.5.15 True False False 160m operator-lifecycle-manager-packageserver 4.5.15 True False False 156m service-ca 4.5.15 True False False 161m storage 4.5.15 True False False 156m [root@OSPAMGR2 4.6.0-rc.2]# oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE openshift-apiserver-operator openshift-apiserver-operator-ddc649644-6bdkt 1/1 Running 2 162m openshift-apiserver apiserver-65b4c9ffd6-5bpqj 1/1 Running 0 159m openshift-apiserver apiserver-65b4c9ffd6-prlwz 1/1 Running 0 159m openshift-apiserver apiserver-65b4c9ffd6-xqqzb 1/1 Running 0 159m openshift-authentication-operator authentication-operator-669844ffdc-bqc6s 1/1 Running 2 162m openshift-authentication oauth-openshift-8c8c8bb8f-bdngx 1/1 Running 0 155m openshift-authentication oauth-openshift-8c8c8bb8f-l8lz9 1/1 Running 0 155m openshift-cloud-credential-operator cloud-credential-operator-c7879d5dd-tppq2 1/1 Running 0 156m openshift-cluster-machine-approver machine-approver-588d44976d-4v55h 2/2 Running 0 162m openshift-cluster-node-tuning-operator cluster-node-tuning-operator-64fd77f697-8tl6z 1/1 Running 0 162m openshift-cluster-node-tuning-operator tuned-6cdwf 1/1 Running 0 161m openshift-cluster-node-tuning-operator tuned-g76gl 1/1 Running 0 161m openshift-cluster-node-tuning-operator tuned-krg4p 1/1 Running 0 156m openshift-cluster-node-tuning-operator tuned-mchv6 1/1 Running 0 161m openshift-cluster-node-tuning-operator tuned-mzqcg 1/1 Running 0 156m openshift-cluster-samples-operator cluster-samples-operator-5d68b5695c-nnl75 2/2 Running 0 155m openshift-cluster-storage-operator cluster-storage-operator-b7f6c797c-cpgdj 1/1 Running 0 156m openshift-cluster-storage-operator csi-snapshot-controller-864c4556cb-5tg6b 1/1 Running 0 156m openshift-cluster-storage-operator csi-snapshot-controller-operator-566775d468-zgbmr 1/1 Running 1 162m openshift-cluster-version cluster-version-operator-96c8b8d88-8r2qc 1/1 Running 0 37m openshift-cluster-version version--bxtts-dpjl8 0/1 Completed 0 38m openshift-config-operator openshift-config-operator-9d4cf66db-zr6tk 1/1 Running 0 155m openshift-console-operator console-operator-5df8c8d4b5-6dz8h 1/1 Running 0 156m openshift-console console-595cc745ff-2pndb 1/1 Running 0 147m openshift-console console-595cc745ff-g98xl 1/1 Running 0 147m openshift-console downloads-547fd9d88d-fwnbr 1/1 Running 0 156m openshift-console downloads-547fd9d88d-ljp7p 1/1 Running 0 156m openshift-controller-manager-operator openshift-controller-manager-operator-54f65787d7-rskbz 1/1 Running 2 162m openshift-controller-manager controller-manager-2sb6h 1/1 Running 0 156m openshift-controller-manager controller-manager-dqsj7 1/1 Running 0 156m openshift-controller-manager controller-manager-tzf2z 1/1 Running 0 156m openshift-dns-operator dns-operator-546f67b8d-9cbfv 2/2 Running 0 162m openshift-dns dns-default-78pjf 3/3 Running 0 160m openshift-dns dns-default-94n69 3/3 Running 0 156m openshift-dns dns-default-qp5q5 3/3 Running 0 160m openshift-dns dns-default-wn9d2 3/3 Running 0 160m openshift-dns dns-default-xc9gr 3/3 Running 0 156m openshift-etcd-operator etcd-operator-7cb94c56-4fjdp 0/1 ImagePullBackOff 0 37m openshift-etcd etcd-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 155m openshift-etcd etcd-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 154m openshift-etcd etcd-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 154m openshift-etcd installer-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-etcd installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-etcd installer-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-etcd installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-etcd installer-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-etcd installer-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-etcd revision-pruner-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-etcd revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-etcd revision-pruner-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-etcd revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-etcd revision-pruner-3-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-etcd revision-pruner-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-image-registry cluster-image-registry-operator-7d9bd9765f-hj4hv 2/2 Running 0 156m openshift-image-registry node-ca-5lzsx 1/1 Running 0 155m openshift-image-registry node-ca-qmsn4 1/1 Running 0 155m openshift-image-registry node-ca-qpqlr 1/1 Running 0 155m openshift-image-registry node-ca-tzghs 1/1 Running 0 155m openshift-image-registry node-ca-wx7wz 1/1 Running 0 155m openshift-ingress-operator ingress-operator-b4948b49c-x47q7 2/2 Running 0 156m openshift-ingress router-default-99598fc9c-6cb55 1/1 Running 0 155m openshift-ingress router-default-99598fc9c-zxfgn 1/1 Running 0 155m openshift-insights insights-operator-859b6f7588-k24qs 1/1 Running 0 156m openshift-kube-apiserver-operator kube-apiserver-operator-5869c75df6-d5pj6 1/1 Running 2 162m openshift-kube-apiserver installer-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-apiserver installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-apiserver installer-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-apiserver installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-apiserver installer-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-apiserver installer-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver installer-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver installer-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 148m openshift-kube-apiserver installer-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 150m openshift-kube-apiserver installer-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-apiserver kube-apiserver-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 148m openshift-kube-apiserver kube-apiserver-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 150m openshift-kube-apiserver kube-apiserver-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m openshift-kube-apiserver revision-pruner-2-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-apiserver revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-apiserver revision-pruner-2-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-apiserver revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-apiserver revision-pruner-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver revision-pruner-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-apiserver revision-pruner-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-apiserver revision-pruner-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 147m openshift-kube-apiserver revision-pruner-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 149m openshift-kube-apiserver revision-pruner-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager-operator kube-controller-manager-operator-9c446d7cc-2q5t7 1/1 Running 2 162m openshift-kube-controller-manager installer-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-controller-manager installer-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager installer-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager installer-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-controller-manager installer-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-controller-manager installer-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-controller-manager installer-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-controller-manager installer-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-controller-manager installer-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager installer-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-controller-manager installer-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-controller-manager kube-controller-manager-master-0.pok-25.pok.stglabs.ibm.com 4/4 Running 0 151m openshift-kube-controller-manager kube-controller-manager-master-1.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m openshift-kube-controller-manager kube-controller-manager-master-2.pok-25.pok.stglabs.ibm.com 4/4 Running 0 152m openshift-kube-controller-manager revision-pruner-3-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager revision-pruner-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-controller-manager revision-pruner-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-controller-manager revision-pruner-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m openshift-kube-controller-manager revision-pruner-5-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-controller-manager revision-pruner-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-controller-manager revision-pruner-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-controller-manager revision-pruner-6-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-controller-manager revision-pruner-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager revision-pruner-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-controller-manager revision-pruner-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-scheduler-operator openshift-kube-scheduler-operator-6674f9c8fd-h87fm 1/1 Running 2 162m openshift-kube-scheduler installer-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 161m openshift-kube-scheduler installer-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-scheduler installer-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler installer-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-scheduler installer-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 159m openshift-kube-scheduler installer-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler installer-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler installer-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-scheduler installer-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 150m openshift-kube-scheduler installer-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-scheduler installer-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 151m openshift-kube-scheduler openshift-kube-scheduler-master-0.pok-25.pok.stglabs.ibm.com 2/2 Running 0 150m openshift-kube-scheduler openshift-kube-scheduler-master-1.pok-25.pok.stglabs.ibm.com 2/2 Running 0 153m openshift-kube-scheduler openshift-kube-scheduler-master-2.pok-25.pok.stglabs.ibm.com 2/2 Running 0 151m openshift-kube-scheduler revision-pruner-2-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-scheduler revision-pruner-3-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 160m openshift-kube-scheduler revision-pruner-4-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler revision-pruner-4-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 156m openshift-kube-scheduler revision-pruner-4-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 158m openshift-kube-scheduler revision-pruner-5-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 155m openshift-kube-scheduler revision-pruner-6-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 154m openshift-kube-scheduler revision-pruner-6-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 153m openshift-kube-scheduler revision-pruner-7-master-0.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 149m openshift-kube-scheduler revision-pruner-7-master-1.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 152m openshift-kube-scheduler revision-pruner-7-master-2.pok-25.pok.stglabs.ibm.com 0/1 Completed 0 150m openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-64d5f49dfc-2zjgl 1/1 Running 2 162m openshift-kube-storage-version-migrator migrator-7f6f887d6c-p58rs 1/1 Running 0 161m openshift-machine-api cluster-autoscaler-operator-845984ffc9-nqf6z 2/2 Running 0 155m openshift-machine-api machine-api-operator-6f8dc9b664-p7hsz 2/2 Running 0 156m openshift-machine-config-operator etcd-quorum-guard-669f5cc5cc-84tlp 1/1 Running 0 160m openshift-machine-config-operator etcd-quorum-guard-669f5cc5cc-mn2h9 1/1 Running 0 160m openshift-machine-config-operator etcd-quorum-guard-669f5cc5cc-r26h4 1/1 Running 0 160m openshift-machine-config-operator machine-config-controller-5cd67cc8fb-6l5sv 1/1 Running 1 161m openshift-machine-config-operator machine-config-daemon-577r5 2/2 Running 0 156m openshift-machine-config-operator machine-config-daemon-77rvm 2/2 Running 0 161m openshift-machine-config-operator machine-config-daemon-j568n 2/2 Running 0 161m openshift-machine-config-operator machine-config-daemon-pfq7w 2/2 Running 0 161m openshift-machine-config-operator machine-config-daemon-zxhp6 2/2 Running 0 156m openshift-machine-config-operator machine-config-operator-68887c4cdf-cjxmd 1/1 Running 1 162m openshift-machine-config-operator machine-config-server-657hv 1/1 Running 0 160m openshift-machine-config-operator machine-config-server-cdm69 1/1 Running 0 160m openshift-machine-config-operator machine-config-server-tjhc5 1/1 Running 0 160m openshift-marketplace certified-operators-69cdf66448-jsz8g 1/1 Running 0 94m openshift-marketplace community-operators-cc799f499-xfgbc 1/1 Running 0 155m openshift-marketplace marketplace-operator-668d865746-7whkk 1/1 Running 0 156m openshift-marketplace redhat-marketplace-d9c7bc85b-fpqsr 1/1 Running 0 155m openshift-marketplace redhat-operators-95f6487c8-qchh2 1/1 Running 0 155m openshift-monitoring alertmanager-main-0 5/5 Running 0 155m openshift-monitoring alertmanager-main-1 5/5 Running 0 155m openshift-monitoring alertmanager-main-2 5/5 Running 0 155m openshift-monitoring cluster-monitoring-operator-55966bbf46-j2nnl 2/2 Running 2 156m openshift-monitoring grafana-67b9fcf78-srgst 2/2 Running 0 155m openshift-monitoring kube-state-metrics-746ddc9785-p4pwc 3/3 Running 0 155m openshift-monitoring node-exporter-jxpqq 2/2 Running 0 155m openshift-monitoring node-exporter-n4gdk 2/2 Running 0 155m openshift-monitoring node-exporter-nsplw 2/2 Running 0 155m openshift-monitoring node-exporter-sjjv5 2/2 Running 0 155m openshift-monitoring node-exporter-tfb7h 2/2 Running 0 155m openshift-monitoring openshift-state-metrics-77cc7ff79f-gsq9c 3/3 Running 0 155m openshift-monitoring prometheus-adapter-85d69fbf6f-b57cv 1/1 Running 0 149m openshift-monitoring prometheus-adapter-85d69fbf6f-xkj69 1/1 Running 0 149m openshift-monitoring prometheus-k8s-0 7/7 Running 1 149m openshift-monitoring prometheus-k8s-1 7/7 Running 1 149m openshift-monitoring prometheus-operator-59bcf8b984-5wc74 2/2 Running 0 155m openshift-monitoring thanos-querier-84455df89d-mrzj8 4/4 Running 0 149m openshift-monitoring thanos-querier-84455df89d-trf7p 4/4 Running 0 149m openshift-multus multus-299nz 1/1 Running 0 162m openshift-multus multus-8wjk6 1/1 Running 0 162m openshift-multus multus-8wlc8 1/1 Running 0 156m openshift-multus multus-admission-controller-bc24q 2/2 Running 0 161m openshift-multus multus-admission-controller-p2tlb 2/2 Running 0 161m openshift-multus multus-admission-controller-v588v 2/2 Running 0 161m openshift-multus multus-fkg5j 1/1 Running 0 162m openshift-multus multus-gngft 1/1 Running 0 156m openshift-network-operator network-operator-6f6657fddb-88nhf 1/1 Running 0 162m openshift-operator-lifecycle-manager catalog-operator-6dd5f7776-gtxjw 1/1 Running 0 162m openshift-operator-lifecycle-manager olm-operator-559cfb8bb6-8gps7 1/1 Running 0 162m openshift-operator-lifecycle-manager packageserver-8d6d667b8-99pdj 1/1 Running 0 160m openshift-operator-lifecycle-manager packageserver-8d6d667b8-p5lbv 1/1 Running 0 160m openshift-sdn ovs-2wkf7 1/1 Running 0 156m openshift-sdn ovs-8857l 1/1 Running 0 156m openshift-sdn ovs-8d594 1/1 Running 0 161m openshift-sdn ovs-9fflj 1/1 Running 0 161m openshift-sdn ovs-jpcqp 1/1 Running 0 161m openshift-sdn sdn-5vs7l 1/1 Running 0 161m openshift-sdn sdn-controller-94qx8 1/1 Running 0 161m openshift-sdn sdn-controller-dbjb2 1/1 Running 0 161m openshift-sdn sdn-controller-jj5fl 1/1 Running 1 161m openshift-sdn sdn-cwpqc 1/1 Running 0 161m openshift-sdn sdn-hkq6v 1/1 Running 0 156m openshift-sdn sdn-nt4wn 1/1 Running 0 161m openshift-sdn sdn-sjl48 1/1 Running 0 156m openshift-service-ca-operator service-ca-operator-66f96555b-wlb29 1/1 Running 2 162m openshift-service-ca service-ca-75b767bcc4-j7wng 1/1 Running 1 161m openshift-service-catalog-removed openshift-service-catalog-apiserver-remover-9665h 0/1 Completed 0 149m openshift-service-catalog-removed openshift-service-catalog-controller-manager-remover-mn6wv 0/1 Completed 0 149m [root@OSPAMGR2 4.6.0-rc.2]# [root@OSPAMGR2 4.6.0-rc.2]# oc describe pod etcd-operator-7cb94c56-4fjdp -n openshift-etcd-operator
Name: etcd-operator-7cb94c56-4fjdp
Namespace: openshift-etcd-operator
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: master-2.pok-25.pok.stglabs.ibm.com/10.20.116.13
Start Time: Wed, 14 Oct 2020 13:38:23 -0400
Labels: app=etcd-operator
pod-template-hash=7cb94c56
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.130.0.49"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.130.0.49"
],
"default": true,
"dns": {}
}]
Status: Pending
IP: 10.130.0.49
IPs:
IP: 10.130.0.49
Controlled By: ReplicaSet/etcd-operator-7cb94c56
Containers:
etcd-operator:
Container ID:
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32
Image ID:
Port: 8443/TCP
Host Port: 0/TCP
Command:
cluster-etcd-operator
operator
Args:
--config=/var/run/configmaps/config/config.yaml
-v=4
--terminate-on-files=/var/run/secrets/serving-cert/tls.crt
--terminate-on-files=/var/run/secrets/serving-cert/tls.key
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Requests:
cpu: 10m
memory: 50Mi
Environment:
IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25ce82e8c5117ccfc7ca6edd84faeadfd72a4298dcb78bfc964e3f633f65837e
OPERATOR_IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32
OPERATOR_IMAGE_VERSION: 4.6.0-rc.4
OPERAND_IMAGE_VERSION: 4.6.0-rc.4
POD_NAME: etcd-operator-7cb94c56-4fjdp (v1:metadata.name)
Mounts:
/var/run/configmaps/config from config (rw)
/var/run/configmaps/etcd-ca from etcd-ca (rw)
/var/run/configmaps/etcd-service-ca from etcd-service-ca (rw)
/var/run/secrets/etcd-client from etcd-client (rw)
/var/run/secrets/kubernetes.io/serviceaccount from etcd-operator-token-qlxwf (ro)
/var/run/secrets/serving-cert from serving-cert (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: etcd-operator-serving-cert
Optional: true
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: etcd-operator-config
Optional: false
etcd-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: etcd-ca-bundle
Optional: false
etcd-service-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: etcd-service-ca-bundle
Optional: false
etcd-client:
Type: Secret (a volume populated by a Secret)
SecretName: etcd-client
Optional: false
etcd-operator-token-qlxwf:
Type: Secret (a volume populated by a Secret)
SecretName: etcd-operator-token-qlxwf
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 120s
node.kubernetes.io/unreachable:NoExecute for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 50m default-scheduler Successfully assigned openshift-etcd-operator/etcd-operator-7cb94c56-4fjdp to master-2.pok-25.pok.stglabs.ibm.com
Normal AddedInterface 50m multus Add eth0 [10.130.0.49/23]
Normal Pulling 49m (x4 over 50m) kubelet Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32"
Warning Failed 49m (x4 over 50m) kubelet Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32": rpc error: code = Unknown desc = (Mirrors also failed: [bastion:5000/ocp4/openshift4@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32: Error reading manifest sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 in bastion:5000/ocp4/openshift4: manifest unknown: manifest unknown]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32: Error reading manifest sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized
Warning Failed 49m (x4 over 50m) kubelet Error: ErrImagePull
Normal BackOff 15m (x152 over 50m) kubelet Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32"
Warning Failed 32s (x217 over 50m) kubelet Error: ImagePullBackOff
[root@OSPAMGR2 4.6.0-rc.2]#
[root@OSPAMGR2 4.6.0-rc.2]# oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32 error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32: unauthorized: access to the requested resource is not authorized [root@OSPAMGR2 4.6.0-rc.2]# [root@OSPAMGR2 /]# oc image info --registry-config='/root/disconnectedinstall/pull-secret.json' quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32
Name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c0a4a1890af3d0e5d06a67a7264bf18c5461fed5bee0da918dc9eb66c518f32
Media Type: application/vnd.docker.distribution.manifest.v2+json
Created: 7d ago
Image Size: 102.2MB in 5 layers
Layers: 71.38MB sha256:024e289f9d3059db8f47e89e0e13314df01f8651dcf9afb50c4c68d58cb92520
1.79kB sha256:34fd41f9801e60f1598818f1a245f68828bf63da35256d5d904a8880d50061aa
4.474MB sha256:cc41e5e3f4a3310408d5cba6603c88bbbc3169dda98ae8ec995044498f9e28ea
10.92MB sha256:a8eac247d57c251bd1da15d4eb49e0589efaf39584419a69348e3aee3ec63d70
15.41MB sha256:c153ba408d51a3f4f4bb1edb66a14cb0e8a4a437d601a153f7d2e45852a1e6be
OS: linux
Arch: s390x
Command: /bin/bash
User: 0
Environment: __doozer=merge
BUILD_RELEASE=202010062159.p0
BUILD_VERSION=v4.6.0
OS_GIT_MAJOR=4
OS_GIT_MINOR=6
OS_GIT_PATCH=0
OS_GIT_TREE_STATE=clean
OS_GIT_VERSION=4.6.0-202010062159.p0-3c9d46e
SOURCE_GIT_TREE_STATE=clean
OS_GIT_COMMIT=3c9d46e
SOURCE_DATE_EPOCH=1601490126
SOURCE_GIT_COMMIT=3c9d46eb6987428dd8ccb65802606eb204737495
SOURCE_GIT_TAG=v0.0.0-alpha.0-572-g3c9d46eb
SOURCE_GIT_URL=https://github.com/openshift/cluster-etcd-operator
GODEBUG=x509ignoreCN=0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=oci
Labels: License=GPLv2+
architecture=s390x
build-date=2020-10-06T22:09:54.796336
com.redhat.build-host=s390-c1-vm-03.prod.osbs.eng.bos.redhat.com
com.redhat.component=cluster-etcd-operator-container
com.redhat.license_terms=https://www.redhat.com/agreements
description=This is the base image from which all OpenShift Container Platform images inherit.
distribution-scope=public
io.k8s.description=This is the base image from which all OpenShift Container Platform images inherit.
io.k8s.display-name=OpenShift Container Platform RHEL 7 Base
io.openshift.build.commit.id=3c9d46eb6987428dd8ccb65802606eb204737495
io.openshift.build.commit.url=https://github.com/openshift/cluster-etcd-operator/commit/3c9d46eb6987428dd8ccb65802606eb204737495
io.openshift.build.source-location=https://github.com/openshift/cluster-etcd-operator
io.openshift.expose-services=
io.openshift.maintainer.component=Etcd Operator
io.openshift.maintainer.product=OpenShift Container Platform
io.openshift.release.operator=true
io.openshift.tags=openshift,base
maintainer=Red Hat, Inc.
name=openshift/ose-cluster-etcd-operator
release=202010062159.p0
summary=Provides the latest release of Red Hat Universal Base Image 8.
url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-cluster-etcd-operator/images/v4.6.0-202010062159.p0
vcs-ref=dd84cdd61cb63a842385f1a00502b032827e297e
vcs-type=git
vendor=Red Hat, Inc.
version=v4.6.0
[root@OSPAMGR2 /]#
Created attachment 1721557 [details]
master-2 kubelet log
oc adm upgrade --force --allow-explicit-upgrade --to-image $target_image Jeremy and Prashanth, Thank you for all your assistance (including with the above posts) with the debug of the disconnected install to connected upgrade for helping us further debug the disconnected install to disconnected upgrade issue. Dynamically changing the cluster pull-secret from disconnected install specific (local mirror registry) to connected install/upgrade specific (quay.io) resolved the issue. Specifically, using the command: oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=authfile where authfile is the updated connected install specific (quay.io) pull-secret file. Thank you, Kyle Thank you Jeremy and Prashanth for your help in debugging our disconnected install to disconnected upgrade issue. As we determined in our meeting, there was a typo on the 'oc adm upgrade' command that we issued where we did not include an "@" before the <sha256_sum_value>. Since we were issuing the upgrade with force, it accepted the command overriding any verifications.
The correct upgrade command that we issued on our test cluster from OCP 4.6 (4.6.0-0.nightly-s390x-2020-10-02-202613) to 4.6 (4.6.0-0.nightly-s390x-2020-10-08-182421) is the following:
# oc adm upgrade --force --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}@sha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1
We may need to highlight the importance of the @ sign in the documentation?
After issuing the correct upgrade command, the "working towards" now shows the percentage complete and is progressing. We are going to fully test this and run several 4.5 to 4.6 (4.6.0-rc.3) upgrades on z/VM and z/KVM. We will update the results here when complete.
Thank you,
-Phil
Jeremy and Prashanth, Thank you again for all your assistance with this issue. Here's an update on the current zVM hosted OCP 4.6 disconnected install upgrade testing. 1. A zVM based OCP 4.6 disconnected install and upgrade from 4.5 (4.5.12) to 4.6 (4.6.0-rc.3) was successful. 2. Additional zVM hosted OCP 4.6 disconnected install upgrade tests are in progress. These tests are using OCP 4.5.14 and 4.5.15 as 4.5 disconnected install bases, upgrading to OCP 4.6.0-rc.3 and 4.6.0-rc.4. We will be following up here with the results of these and other additional OCP 4.6 disconnected install and upgrade tests results for zKVM and zVM hosted clusters later today. Thank you, Kyle Jeremy and Prashanth, Just a quick reminder to open a new bugzilla (or we can, whichever is your preference) for the OCP 4.5 disconnected install "oc adm must-gather" issue where when running this command on a OCP 4.5 disconnected installed cluster it fails, as opposed to a OCP 4.5 connected installed cluster where the "oc adm must-gather" command succeeds. 1. Please see comment #5 above, specifically the error message from the "oc adm must-gather" command which appears to be an x86 module loaded instead of an s390x module: standard_init_linux.go:210: exec user process caused "exec format error" 2. We have tested further and have seen this same issue on OCP 4.5.12, 4.5.13, 4.5.14, and 4.5.15 for disconnected installs only, and not with connected installs. Thank you, Kyle Kyle/Phil, Thanks! If it is ok with you , I am going to close this bug. Could you also open a new BZ for them must gather issue and provide this information in that BZ: - run an `oc image info <must-gather-image>` where the must-gather-image is the mirrored must gather image. This will give us an idea of what got mirrored. Thanks Prashanth I've filed the new issue here: https://bugzilla.redhat.com/show_bug.cgi?id=1888738 (In reply to Philip Chan from comment #15) > Thank you Jeremy and Prashanth for your help in debugging our disconnected > install to disconnected upgrade issue. As we determined in our meeting, > there was a typo on the 'oc adm upgrade' command that we issued where we did > not include an "@" before the <sha256_sum_value>. Can you paste the command you ran with the incorrect pullspec? I'd expect us to do a better job complaining about pullspecs that lacked a digest (e.g. via [1]). [1]: https://github.com/openshift/oc/pull/390 Hi,
The incorrect command that was issued when attempting to upgrade to 4.6.0-0.nightly-s390x-2020-10-08-18242 was the following:
# oc adm upgrade --force --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}sha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1
Thanks,
Phil
Checking with a recent-ish client: $ oc version --client Client Version: 4.5.11 $ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-releasesha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1 error: --to-image must be a by-digest pull spec, unless --force is also set, because release images that are not accessed via digest cannot be verified by the cluster. Even when --force is set, using tags is not recommended, although we continue to allow it for backwards compatibility So that's good. But you point out that you used --force, the by-tag pullspec becomes a non-fatal stderr warning: $ oc adm upgrade --force --to-image quay.io/openshift-release-dev/ocp-releasesha256:9be577ca37cf8aa5c9b97b33a6cc1551bdbad4dc6fdfd628b0a364bce35eb7d1 warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead Error from server (Forbidden): clusterversions.config.openshift.io "version" is forbidden: User "wking" cannot get clusterversions.config.openshift.io at the cluster scope: no RBAC policy matched The failure there is because I'm not an admin on the cluster I'm authenticating against, but it shows the logged pullspec warning. Also not clear to me why you are using --force. Are you not mirroring the signature as a ConfigMap [1]? Official docs around this in [2] need some polish still, but the procedure in [1] is what you want for sneaker-net mirroring. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1837675#c5 [2]: https://docs.openshift.com/container-platform/4.5/updating/updating-restricted-network-cluster.html#updating-restricted-network-image-signature-configmap Jeremy and Prashanth,
The following zVM hosted OCP 4.6 cluster disconnected install builds were successfully upgraded with the corresponding disconnected upgrade builds.
Disconnected install build Disconnected upgrade build
========================== ==========================
1. 4.5.4 4.6.0-rc.3
2. 4.5.4 4.6.0-rc.4
3. 4.5.5 4.6.0-rc.3
4. 4.5.5 4.6.0-rc.4
5. 4.5.12 4.6.0-rc.3
6. 4.5.12 4.6.0-rc.4
7. 4.5.14 4.6.0-rc.3
8. 4.5.14 4.6.0-rc.4
9. 4.5.15 4.6.0-rc.3
10. 4.5.15 4.6.0-rc.4
Thank you,
Kyle
Hi Trevor,
That is correct, we do not need to use the --force for the upgrade. The --force was left as a result of our debug (4.6.x -> 4.6.x). I just ran a quick upgrade test without the --force and that succeeded:
# oc adm upgrade --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}@sha256:2d286c599fd767ea5041c9e0b11863efa321daaf9616d640199601e8dd9da3d4
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image bastion:5000/ocp4/openshift4@sha256:2d286c599fd767ea5041c9e0b11863efa321daaf9616d640199601e8dd9da3d4
With regards to the ConfigMap, I agree that we need some work with the official documentation. I passed in the --apply-release-image-signature option when performing the mirroring, as documented, if that is done, we did not perform the ConfigMap steps. Here's what we passed:
# oc adm -a ${LOCAL_SECRET_JSON} release mirror --from=quay.io/openshift-release-dev/ocp-release@sha256:2d286c599fd767ea5041c9e0b11863efa321daaf9616d640199601e8dd9da3d4 --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE} --apply-release-image-signature
Hope that info helps, please let me know if you need any additional info.
Thanks,
Phil
|