Bug 1855479
| Summary: | Post-upgrade the MCO failing to evict "build" pods during drain | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | kube-apiserver | Assignee: | Stefan Schimanski <sttts> |
| Status: | CLOSED ERRATA | QA Contact: | Ke Wang <kewang> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.5 | CC: | aos-bugs, ccoleman, deads, dwalsh, jokerman, lmohanty, mfojtik, mgugino, mpatel, rphillips, scuppett, wking, xxia |
| Target Milestone: | --- | Keywords: | Upgrades |
| Target Release: | 4.3.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-23 13:52:39 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1855478 | ||
| Bug Blocks: | |||
|
Comment 1
W. Trevor King
2020-07-10 01:24:06 UTC
Ryan is on leave - Tried upgrade 4.2.z to 4.3 nightly build,
IPI install on AWS, upgrade from 4.2.36 to 4.3 nightly 4.3.0-0.nightly-2020-09-09-043522 succeeded with builds pods getting terminated.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.2.36 True False 6m22s Cluster version is 4.2.36
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-131-241.us-east-2.compute.internal Ready master 19m v1.14.6-152-g117ba1f 10.0.131.241 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-143-227.us-east-2.compute.internal Ready worker 13m v1.14.6-152-g117ba1f 10.0.143.227 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-157-23.us-east-2.compute.internal Ready master 19m v1.14.6-152-g117ba1f 10.0.157.23 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-159-130.us-east-2.compute.internal Ready worker 13m v1.14.6-152-g117ba1f 10.0.159.130 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-160-238.us-east-2.compute.internal Ready master 19m v1.14.6-152-g117ba1f 10.0.160.238 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-164-122.us-east-2.compute.internal Ready worker 13m v1.14.6-152-g117ba1f 10.0.164.122 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8_2
$ cat newapp.sh
#!/usr/bin/env bash
newapp(){
oc new-app ./ruby-hello-world/ --strategy=docker
oc new-app openshift/ruby:2.5~https://github.com/wangke19/ruby-ex
oc new-app https://github.com/sclorg/cakephp-ex
oc new-app https://github.com/sclorg/nodejs-ex
}
for i in {1..10}; do
oc new-project test$i
if [ $i -eq 9 -o $i -eq 10 ];then
oc adm policy add-scc-to-user privileged system:serviceaccount:test$i:builder
fi
newapp
done
Using above script to create some apps.
$ oc get pods -A | grep ^test
test1 cakephp-ex-1-build 0/1 Completed 0 23m
test1 cakephp-ex-6b884dd687-jdtdx 1/1 Running 0 19m
test1 nodejs-ex-1-build 0/1 Completed 0 22m
test1 nodejs-ex-7bdf958f7f-f4dl5 1/1 Running 0 18m
test1 ruby-ex-1-build 0/1 Completed 0 23m
test1 ruby-ex-7577b4dd55-fpdtt 1/1 Running 0 20m
test1 ruby-hello-world-1-build 0/1 Completed 0 23m
test1 ruby-hello-world-59589497-bbxxf 1/1 Running 0 21m
test10 cakephp-ex-1-build 0/1 Completed 0 15m
test10 cakephp-ex-6d77b74995-5jfkt 1/1 Running 0 5m7s
test10 nodejs-ex-1-build 0/1 OOMKilled 0 15m
test10 nodejs-ex-7698c47fd6-zggf2 1/1 Running 0 9m8s
test10 ruby-ex-1-build 0/1 Completed 0 15m
test10 ruby-ex-7bc9777745-zhzhf 1/1 Running 0 5m51s
test10 ruby-hello-world-1-build 0/1 Completed 0 15m
test10 ruby-hello-world-54bc4b48df-zmzft 1/1 Running 0 5m2s
test2 cakephp-ex-1-build 0/1 Completed 0 22m
test2 cakephp-ex-5dd565f6b6-wb67m 1/1 Running 0 17m
test2 nodejs-ex-1-build 0/1 Completed 0 21m
test2 nodejs-ex-6cc88874bb-tsqqn 1/1 Running 0 14m
test2 ruby-ex-1-build 0/1 Completed 0 22m
test2 ruby-ex-747bcdb76c-j4pkp 1/1 Running 0 18m
test2 ruby-hello-world-1-build 0/1 Completed 0 22m
test2 ruby-hello-world-6f7f4ccf98-6cbrw 1/1 Running 0 16m
test3 cakephp-ex-1-build 0/1 Completed 0 21m
test3 cakephp-ex-657f7bf957-wnds5 1/1 Running 0 13m
test3 nodejs-ex-1-build 0/1 OOMKilled 0 21m
test3 nodejs-ex-8b86c7d86-862ck 1/1 Running 0 16m
test3 ruby-ex-1-build 0/1 Completed 0 21m
test3 ruby-ex-7b77499c6-54d7h 1/1 Running 0 13m
test3 ruby-hello-world-1-build 0/1 Completed 0 21m
test3 ruby-hello-world-848d6fffcc-fr2k8 1/1 Running 0 14m
test4 cakephp-ex-1-build 0/1 Completed 0 20m
test4 cakephp-ex-76f579894c-s6x7l 1/1 Running 0 11m
test4 nodejs-ex-1-build 0/1 Completed 0 20m
test4 nodejs-ex-757fb7f8c8-l46dj 1/1 Running 0 11m
test4 ruby-ex-1-build 0/1 Completed 0 20m
test4 ruby-ex-54bf97bbcb-sp49w 1/1 Running 0 11m
test4 ruby-hello-world-1-build 0/1 Completed 0 20m
test4 ruby-hello-world-7c9bdcfdf5-gnjpt 1/1 Running 0 12m
test5 cakephp-ex-1-build 0/1 Completed 0 19m
test5 cakephp-ex-57c877986b-6r9fp 1/1 Running 0 11m
test5 nodejs-ex-1-build 0/1 Completed 0 19m
test5 nodejs-ex-c7c5b667b-srm26 1/1 Running 0 11m
test5 ruby-ex-1-build 0/1 Completed 0 19m
test5 ruby-ex-8684b74769-5x62c 1/1 Running 0 11m
test5 ruby-hello-world-1-build 0/1 Completed 0 20m
test5 ruby-hello-world-7c8d887577-zb2m6 1/1 Running 0 11m
test6 cakephp-ex-1-build 0/1 Completed 0 18m
test6 cakephp-ex-698c7989c8-t7cqm 1/1 Running 0 9m15s
test6 nodejs-ex-1-build 0/1 Completed 0 18m
test6 nodejs-ex-86dfcb8c48-qpn65 1/1 Running 0 10m
test6 ruby-ex-1-build 0/1 Completed 0 19m
test6 ruby-ex-769798c47-km4kc 1/1 Running 0 11m
test6 ruby-hello-world-1-build 0/1 Completed 0 19m
test6 ruby-hello-world-669fc65c-kgtqv 1/1 Running 0 7m34s
test7 cakephp-ex-1-build 0/1 Completed 0 18m
test7 cakephp-ex-59c5b8d69c-w2pk4 1/1 Running 0 6m10s
test7 nodejs-ex-1-build 0/1 Completed 0 17m
test7 nodejs-ex-6bb9678bc4-r8zj7 1/1 Running 0 9m21s
test7 ruby-ex-1-build 0/1 Completed 0 18m
test7 ruby-ex-6c8c98f556-b88d8 1/1 Running 0 10m
test7 ruby-hello-world-1-build 0/1 Completed 0 18m
test7 ruby-hello-world-db8c8569b-zbr67 1/1 Running 0 6m12s
test8 cakephp-ex-1-build 0/1 Completed 0 17m
test8 cakephp-ex-d6bd47599-cpqpc 1/1 Running 0 9m46s
test8 nodejs-ex-1-build 0/1 Completed 0 17m
test8 nodejs-ex-74b9c9b6b7-pwsxx 1/1 Running 0 6m38s
test8 ruby-ex-1-build 0/1 Completed 0 17m
test8 ruby-ex-867497b6dd-g7ksm 1/1 Running 0 9m3s
test8 ruby-hello-world-1-build 0/1 Completed 0 17m
test8 ruby-hello-world-79797cffff-9kf7p 1/1 Running 0 5m52s
test9 cakephp-ex-1-build 0/1 Completed 0 16m
test9 cakephp-ex-857b798b57-cl77n 1/1 Running 0 9m47s
test9 nodejs-ex-1-build 0/1 Completed 0 16m
test9 nodejs-ex-67fcb884c9-bx48m 1/1 Running 0 6m2s
test9 ruby-ex-1-build 0/1 Completed 0 16m
test9 ruby-ex-55b8bf94c5-j6sk7 1/1 Running 0 6m41s
test9 ruby-hello-world-1-build 0/1 Completed 0 16m
test9 ruby-hello-world-7bb5858cdd-rkj9t 1/1 Running 0 8m40s
$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-09-09-043522 --allow-explicit-upgrade --force
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.3.0-0.nightly-2020-09-09-043522 True False 31m Cluster version is 4.3.0-0.nightly-2020-09-09-043522
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT
master rendered-master-59c6331d1421bfabeffbcd20bf8df0de True False False 3 3 3 0
worker rendered-worker-26925141dedf82d1c1313502a244a20a True False False 3 3 3 0
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.3.0-0.nightly-2020-09-09-043522 True False False 127m
cloud-credential 4.3.0-0.nightly-2020-09-09-043522 True False False 138m
cluster-autoscaler 4.3.0-0.nightly-2020-09-09-043522 True False False 131m
console 4.3.0-0.nightly-2020-09-09-043522 True False False 36m
dns 4.3.0-0.nightly-2020-09-09-043522 True False False 137m
image-registry 4.3.0-0.nightly-2020-09-09-043522 True False False 45m
ingress 4.3.0-0.nightly-2020-09-09-043522 True False False 39m
insights 4.3.0-0.nightly-2020-09-09-043522 True False False 138m
kube-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 135m
kube-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 135m
kube-scheduler 4.3.0-0.nightly-2020-09-09-043522 True False False 135m
machine-api 4.3.0-0.nightly-2020-09-09-043522 True False False 138m
machine-config 4.3.0-0.nightly-2020-09-09-043522 True False False 137m
marketplace 4.3.0-0.nightly-2020-09-09-043522 True False False 37m
monitoring 4.3.0-0.nightly-2020-09-09-043522 True False False 45m
network 4.3.0-0.nightly-2020-09-09-043522 True False False 137m
node-tuning 4.3.0-0.nightly-2020-09-09-043522 True False False 38m
openshift-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 35m
openshift-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 136m
openshift-samples 4.3.0-0.nightly-2020-09-09-043522 True False False 55m
operator-lifecycle-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 134m
operator-lifecycle-manager-catalog 4.3.0-0.nightly-2020-09-09-043522 True False False 134m
operator-lifecycle-manager-packageserver 4.3.0-0.nightly-2020-09-09-043522 True False False 38m
service-ca 4.3.0-0.nightly-2020-09-09-043522 True False False 138m
service-catalog-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 134m
service-catalog-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 134m
storage 4.3.0-0.nightly-2020-09-09-043522 True False False 69m
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-131-241.us-east-2.compute.internal Ready master 140m v1.16.2+417b9fd 10.0.131.241 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-143-227.us-east-2.compute.internal Ready worker 134m v1.16.2+417b9fd 10.0.143.227 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-157-23.us-east-2.compute.internal Ready master 140m v1.16.2+417b9fd 10.0.157.23 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-159-130.us-east-2.compute.internal Ready worker 134m v1.16.2+417b9fd 10.0.159.130 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-160-238.us-east-2.compute.internal Ready master 140m v1.16.2+417b9fd 10.0.160.238 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-164-122.us-east-2.compute.internal Ready worker 134m v1.16.2+417b9fd 10.0.164.122 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
$ oc get pods -A | grep ^test
test1 cakephp-ex-6b884dd687-ncd9m 1/1 Running 0 46m
test1 nodejs-ex-7bdf958f7f-kh9lj 1/1 Running 0 46m
test1 ruby-ex-7577b4dd55-z6bqj 1/1 Running 0 40m
test1 ruby-hello-world-59589497-nqz9k 1/1 Running 0 40m
test10 cakephp-ex-6d77b74995-zxxpw 1/1 Running 0 46m
test10 nodejs-ex-7698c47fd6-4598c 1/1 Running 0 40m
test10 ruby-ex-7bc9777745-mh9b5 1/1 Running 0 46m
test10 ruby-hello-world-54bc4b48df-q9pbz 1/1 Running 0 40m
test2 cakephp-ex-5dd565f6b6-6qbnw 1/1 Running 0 40m
test2 nodejs-ex-6cc88874bb-9ptzz 1/1 Running 0 40m
test2 ruby-ex-747bcdb76c-8lp6r 1/1 Running 0 46m
test2 ruby-hello-world-6f7f4ccf98-mpdxt 1/1 Running 0 40m
test3 cakephp-ex-657f7bf957-2b25d 1/1 Running 0 40m
test3 nodejs-ex-8b86c7d86-xvjfd 1/1 Running 0 40m
test3 ruby-ex-7b77499c6-rwdbl 1/1 Running 0 46m
test3 ruby-hello-world-848d6fffcc-7zq6b 1/1 Running 0 40m
test4 cakephp-ex-76f579894c-lhxvg 1/1 Running 0 40m
test4 nodejs-ex-757fb7f8c8-k8hmm 1/1 Running 0 46m
test4 ruby-ex-54bf97bbcb-pddpj 1/1 Running 0 46m
test4 ruby-hello-world-7c9bdcfdf5-rbhv4 1/1 Running 0 46m
test5 cakephp-ex-57c877986b-78vdg 1/1 Running 0 40m
test5 nodejs-ex-c7c5b667b-7zkch 1/1 Running 0 46m
test5 ruby-ex-8684b74769-qqpv5 1/1 Running 0 40m
test5 ruby-hello-world-7c8d887577-c5l5c 1/1 Running 0 46m
test6 cakephp-ex-698c7989c8-gc9wd 1/1 Running 0 46m
test6 nodejs-ex-86dfcb8c48-kbn78 1/1 Running 0 46m
test6 ruby-ex-769798c47-598nw 1/1 Running 0 40m
test6 ruby-hello-world-669fc65c-km8mm 1/1 Running 0 46m
test7 cakephp-ex-59c5b8d69c-8jd7c 1/1 Running 0 46m
test7 nodejs-ex-6bb9678bc4-j5n5h 1/1 Running 0 40m
test7 ruby-ex-6c8c98f556-5ptbk 1/1 Running 0 40m
test7 ruby-hello-world-db8c8569b-4zk9l 1/1 Running 0 40m
test8 cakephp-ex-d6bd47599-j9slp 1/1 Running 0 40m
test8 nodejs-ex-74b9c9b6b7-vw5z8 1/1 Running 0 40m
test8 ruby-ex-867497b6dd-bpp2l 1/1 Running 0 46m
test8 ruby-hello-world-79797cffff-l4pmp 1/1 Running 0 46m
test9 cakephp-ex-857b798b57-jk8zl 1/1 Running 0 40m
test9 nodejs-ex-67fcb884c9-mk95p 1/1 Running 0 40m
test9 ruby-ex-55b8bf94c5-47zxj 1/1 Running 0 40m
test9 ruby-hello-world-7bb5858cdd-9dxxm 1/1 Running 0 46m
- Tried upgrade between 4.3.z.
IPI install on GCP, upgrade from 4.3.35 to 4.3 nightly 4.3.0-0.nightly-2020-09-09-043522 succeeded with builds pods getting terminated.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.3.35 True False 71m Cluster version is 4.3.35
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kewang-p6sw8-m-0.c.openshift-qe.internal Ready master 93m v1.16.2+7279a4a 10.0.0.5 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-1.c.openshift-qe.internal Ready master 94m v1.16.2+7279a4a 10.0.0.3 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-2.c.openshift-qe.internal Ready master 93m v1.16.2+7279a4a 10.0.0.4 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-a-njvk2.c.openshift-qe.internal Ready worker 80m v1.16.2+7279a4a 10.0.32.2 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-b-g75qq.c.openshift-qe.internal Ready worker 80m v1.16.2+7279a4a 10.0.32.3 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-c-m2b4n.c.openshift-qe.internal Ready worker 80m v1.16.2+7279a4a 10.0.32.4 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
$ cat newapp.sh
#!/usr/bin/env bash
newapp(){
oc new-app ./ruby-hello-world/ --strategy=docker
oc new-app openshift/ruby:2.5~https://github.com/wangke19/ruby-ex
oc new-app https://github.com/sclorg/cakephp-ex
oc new-app https://github.com/sclorg/nodejs-ex
}
for i in {1..10}; do
oc new-project test$i
if [ $i -eq 9 -o $i -eq 10 ];then
oc adm policy add-scc-to-user privileged system:serviceaccount:test$i:builder
fi
newapp
done
Using above script to create some apps.
$ oc get pods -A | grep ^test
test1 cakephp-ex-1-build 0/1 Completed 0 18m
test1 cakephp-ex-5669cf9595-9gd22 1/1 Running 0 16m
test1 nodejs-ex-1-build 0/1 Completed 0 18m
test1 nodejs-ex-bc9dc8559-2xl2s 1/1 Running 0 17m
test1 ruby-ex-1-build 0/1 Completed 0 19m
test1 ruby-ex-7857965989-6gnf8 1/1 Running 0 18m
test1 ruby-hello-world-1-build 0/1 Completed 0 19m
test1 ruby-hello-world-7797d99584-pjtmg 1/1 Running 0 17m
test10 cakephp-ex-1-build 0/1 Completed 0 5m36s
test10 cakephp-ex-57db887d6c-4h5kv 1/1 Running 0 4m14s
test10 nodejs-ex-1-build 0/1 Completed 0 5m13s
test10 nodejs-ex-67cd8b4844-6mpqb 1/1 Running 0 4m29s
test10 ruby-ex-1-build 0/1 Completed 0 6m2s
test10 ruby-ex-7476bfb668-wjrxs 1/1 Running 0 5m18s
test10 ruby-hello-world-1-build 0/1 Completed 0 6m13s
test10 ruby-hello-world-b4595bd64-zvx66 1/1 Running 0 5m5s
test2 cakephp-ex-1-build 0/1 Completed 0 17m
test2 cakephp-ex-dfd477ddd-c256r 1/1 Running 0 15m
test2 nodejs-ex-1-build 0/1 Completed 0 16m
test2 nodejs-ex-656f966b5b-p8wmq 1/1 Running 0 15m
test2 ruby-ex-1-build 0/1 Completed 0 17m
test2 ruby-ex-684bfc68df-ctmqj 1/1 Running 0 16m
test2 ruby-hello-world-1-build 0/1 Completed 0 17m
test2 ruby-hello-world-6c7d567c7d-87d6z 1/1 Running 0 16m
test3 cakephp-ex-1-build 0/1 Completed 0 15m
test3 cakephp-ex-5958fc74-pdx9k 1/1 Running 0 13m
test3 nodejs-ex-1-build 0/1 Completed 0 15m
test3 nodejs-ex-dbbcfc67f-n9j27 1/1 Running 0 14m
test3 ruby-ex-1-build 0/1 Completed 0 16m
test3 ruby-ex-69b7787699-5npfp 1/1 Running 0 15m
test3 ruby-hello-world-1-build 0/1 Completed 0 16m
test3 ruby-hello-world-86995d466b-wbm5q 1/1 Running 0 14m
test4 cakephp-ex-1-build 0/1 Completed 0 14m
test4 cakephp-ex-6dfd74d678-9zjwn 1/1 Running 0 12m
test4 nodejs-ex-1-build 0/1 Completed 0 14m
test4 nodejs-ex-7546c5bc8-d8xjj 1/1 Running 0 12m
test4 ruby-ex-1-build 0/1 Completed 0 15m
test4 ruby-ex-66cddc9d5-bbmgg 1/1 Running 0 14m
test4 ruby-hello-world-1-build 0/1 Completed 0 15m
test4 ruby-hello-world-6676457779-hwpxf 1/1 Running 0 13m
test5 cakephp-ex-1-build 0/1 Completed 0 13m
test5 cakephp-ex-5486f85f84-dvl96 1/1 Running 0 11m
test5 nodejs-ex-1-build 0/1 Completed 0 12m
test5 nodejs-ex-858cd79ff5-5r79z 1/1 Running 0 11m
test5 ruby-ex-1-build 0/1 Completed 0 13m
test5 ruby-ex-f548f5f5f-pfhkt 1/1 Running 0 12m
test5 ruby-hello-world-1-build 0/1 Completed 0 13m
test5 ruby-hello-world-5dd5db7576-mdvpp 1/1 Running 0 12m
test6 cakephp-ex-1-build 0/1 Completed 0 11m
test6 cakephp-ex-84cf68774f-62d84 1/1 Running 0 10m
test6 nodejs-ex-1-build 0/1 Completed 0 11m
test6 nodejs-ex-79bb7fc4fb-g4cjl 1/1 Running 0 10m
test6 ruby-ex-1-build 0/1 Completed 0 12m
test6 ruby-ex-544496f6dc-r962v 1/1 Running 0 11m
test6 ruby-hello-world-1-build 0/1 Completed 0 12m
test6 ruby-hello-world-6488bd9dd8-5xlzb 1/1 Running 0 11m
test7 cakephp-ex-1-build 0/1 Completed 0 9m57s
test7 cakephp-ex-7cf8d866c9-dpdtf 1/1 Running 0 8m33s
test7 nodejs-ex-1-build 0/1 Completed 0 9m32s
test7 nodejs-ex-844d6ccf64-h84l8 1/1 Running 0 8m39s
test7 ruby-ex-1-build 0/1 Completed 0 10m
test7 ruby-ex-5544c8879-n6kgk 1/1 Running 0 9m43s
test7 ruby-hello-world-1-build 0/1 Completed 0 10m
test7 ruby-hello-world-74594b4675-t5mjf 1/1 Running 0 9m52s
test8 cakephp-ex-1-build 0/1 Completed 0 8m29s
test8 cakephp-ex-7995f78697-c7v7d 1/1 Running 0 7m10s
test8 nodejs-ex-1-build 0/1 Completed 0 8m8s
test8 nodejs-ex-7b5c94ff88-gt25v 1/1 Running 0 7m7s
test8 ruby-ex-1-build 0/1 Completed 0 8m56s
test8 ruby-ex-59f8c8f748-rg4jz 1/1 Running 0 7m58s
test8 ruby-hello-world-1-build 0/1 Completed 0 9m8s
test8 ruby-hello-world-79d47b4d-trjgs 1/1 Running 0 7m50s
test9 cakephp-ex-1-build 0/1 Completed 0 7m7s
test9 cakephp-ex-56945f9ff-mvp4b 1/1 Running 0 5m24s
test9 nodejs-ex-1-build 0/1 Completed 0 6m42s
test9 nodejs-ex-64ff55cf59-8mrm8 1/1 Running 0 5m42s
test9 ruby-ex-1-build 0/1 Completed 0 7m29s
test9 ruby-ex-7bb4bfdd8f-gktr9 1/1 Running 0 6m9s
test9 ruby-hello-world-1-build 0/1 Completed 0 7m40s
test9 ruby-hello-world-86d76db4dd-9xzkn 1/1 Running 0 6m41s
$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-09-09-043522 --allow-explicit-upgrade --force
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.3.0-0.nightly-2020-09-09-043522 True False 3h5m Cluster version is 4.3.0-0.nightly-2020-09-09-043522
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT
master rendered-master-cbcdc8e72e97bbe782402cf26c204fbe True False False 3 3 3 0
worker rendered-worker-eeda4a8846152a8c46b17dbf377c8184 True False False 3 3 3 0
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.3.0-0.nightly-2020-09-09-043522 True False False 5h16m
cloud-credential 4.3.0-0.nightly-2020-09-09-043522 True False False 5h38m
cluster-autoscaler 4.3.0-0.nightly-2020-09-09-043522 True False False 5h26m
console 4.3.0-0.nightly-2020-09-09-043522 True False False 3h10m
dns 4.3.0-0.nightly-2020-09-09-043522 True False False 5h36m
image-registry 4.3.0-0.nightly-2020-09-09-043522 True False False 5h23m
ingress 4.3.0-0.nightly-2020-09-09-043522 True False False 3h13m
insights 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m
kube-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 5h35m
kube-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h34m
kube-scheduler 4.3.0-0.nightly-2020-09-09-043522 True False False 5h34m
machine-api 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m
machine-config 4.3.0-0.nightly-2020-09-09-043522 True False False 3h7m
marketplace 4.3.0-0.nightly-2020-09-09-043522 True False False 3h9m
monitoring 4.3.0-0.nightly-2020-09-09-043522 True False False 5h17m
network 4.3.0-0.nightly-2020-09-09-043522 True False False 5h36m
node-tuning 4.3.0-0.nightly-2020-09-09-043522 True False False 3h9m
openshift-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 3h7m
openshift-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h35m
openshift-samples 4.3.0-0.nightly-2020-09-09-043522 True False False 3h33m
operator-lifecycle-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m
operator-lifecycle-manager-catalog 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m
operator-lifecycle-manager-packageserver 4.3.0-0.nightly-2020-09-09-043522 True False False 3h9m
service-ca 4.3.0-0.nightly-2020-09-09-043522 True False False 5h36m
service-catalog-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m
service-catalog-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m
storage 4.3.0-0.nightly-2020-09-09-043522 True False False 3h33m
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kewang-p6sw8-m-0.c.openshift-qe.internal Ready master 5h38m v1.16.2+417b9fd 10.0.0.5 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-1.c.openshift-qe.internal Ready master 5h38m v1.16.2+417b9fd 10.0.0.3 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-2.c.openshift-qe.internal Ready master 5h38m v1.16.2+417b9fd 10.0.0.4 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-a-njvk2.c.openshift-qe.internal Ready worker 5h24m v1.16.2+417b9fd 10.0.32.2 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-b-g75qq.c.openshift-qe.internal Ready worker 5h25m v1.16.2+417b9fd 10.0.32.3 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-c-m2b4n.c.openshift-qe.internal Ready worker 5h24m v1.16.2+417b9fd 10.0.32.4 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8
$ oc get pods -A | grep ^test
test1 cakephp-ex-5669cf9595-rctd4 1/1 Running 0 3h14m
test1 nodejs-ex-bc9dc8559-h55fh 1/1 Running 0 3h14m
test1 ruby-ex-7857965989-9lk4t 1/1 Running 0 3h14m
test1 ruby-hello-world-7797d99584-4brgs 1/1 Running 0 3h14m
test10 cakephp-ex-57db887d6c-j8p2h 1/1 Running 0 3h16m
test10 nodejs-ex-67cd8b4844-4wk82 1/1 Running 0 3h14m
test10 ruby-ex-7476bfb668-7g5zw 1/1 Running 0 3h14m
test10 ruby-hello-world-b4595bd64-lkbnx 1/1 Running 0 3h14m
test2 cakephp-ex-dfd477ddd-m7n4v 1/1 Running 0 3h14m
test2 nodejs-ex-656f966b5b-bmzkf 1/1 Running 0 3h14m
test2 ruby-ex-684bfc68df-l6jbn 1/1 Running 0 3h14m
test2 ruby-hello-world-6c7d567c7d-mn7tx 1/1 Running 0 3h16m
test3 cakephp-ex-5958fc74-pqb5x 1/1 Running 0 3h16m
test3 nodejs-ex-dbbcfc67f-rlrvf 1/1 Running 0 3h14m
test3 ruby-ex-69b7787699-w2r8h 1/1 Running 0 3h16m
test3 ruby-hello-world-86995d466b-l6p4d 1/1 Running 0 3h16m
test4 cakephp-ex-6dfd74d678-2fbwc 1/1 Running 0 3h14m
test4 nodejs-ex-7546c5bc8-2pkzh 1/1 Running 0 3h14m
test4 ruby-ex-66cddc9d5-57n7c 1/1 Running 0 3h16m
test4 ruby-hello-world-6676457779-vhxqt 1/1 Running 0 3h16m
test5 cakephp-ex-5486f85f84-fph8x 1/1 Running 0 3h14m
test5 nodejs-ex-858cd79ff5-8g828 1/1 Running 0 3h16m
test5 ruby-ex-f548f5f5f-5f79v 1/1 Running 0 3h16m
test5 ruby-hello-world-5dd5db7576-ztfwq 1/1 Running 0 3h16m
test6 cakephp-ex-84cf68774f-dg67t 1/1 Running 0 3h14m
test6 nodejs-ex-79bb7fc4fb-zb9lx 1/1 Running 0 3h16m
test6 ruby-ex-544496f6dc-lfwjp 1/1 Running 0 3h14m
test6 ruby-hello-world-6488bd9dd8-tzq76 1/1 Running 0 3h14m
test7 cakephp-ex-7cf8d866c9-mc78k 1/1 Running 0 3h16m
test7 nodejs-ex-844d6ccf64-9zssg 1/1 Running 0 3h14m
test7 ruby-ex-5544c8879-qm459 1/1 Running 0 3h14m
test7 ruby-hello-world-74594b4675-5nc5d 1/1 Running 0 3h16m
test8 cakephp-ex-7995f78697-tjwbw 1/1 Running 0 3h16m
test8 nodejs-ex-7b5c94ff88-4z8rt 1/1 Running 0 3h16m
test8 ruby-ex-59f8c8f748-r9pnn 1/1 Running 0 3h16m
test8 ruby-hello-world-79d47b4d-98lxq 1/1 Running 0 3h14m
test9 cakephp-ex-56945f9ff-phpms 1/1 Running 0 3h14m
test9 nodejs-ex-64ff55cf59-jkrvs 1/1 Running 0 3h16m
test9 ruby-ex-7bb4bfdd8f-sh68t 1/1 Running 0 3h16m
test9 ruby-hello-world-86d76db4dd-gwqg9 1/1 Running 0 3h16m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.3.38 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3609 |